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SUMMARY 



The aims of this study were to identify, select, analyze, and 
describe educational programs for culturally disadvantaged children 
from preschool through grade 12 which had yielded measured benefits 
of cognitive achievement. 

The written reports of over 1,000 compensatory educational pro- 
grams were perused in a literature search carried out mainly through 
Educational Resources Information Centers (ERIC), libraries, and some 
300 mail requests. With the assistance of a panel of national experts, 
a list was compiled of about 100 programs, selected not on the basis 
of geographical area, grade level, or type of treatment, but because 
each. was believed to have enabled its pupils to make greater gains in 
measured cognitive achievement than they normally would have made had 
they not received the program. The list was restricted also by con- 
sidering only programs reported between 1963 and 1968. 

A schedule of site visits was established and carried out, to 
include eventually 98 programs in 31 urban areas and 16 states. 

During the site visits, structured interviews were held to obtain all 
data necessary to decide whether the program under study had indeed 
provided measured benefits of cognitive achievement, and if it had, 
to compile a complete description and to conduct an analysis of the 
components of the successful programs. 

The site visits were conducted by five senior staff of the 
American Institutes for Research, with assistance from other staff 
on two occasions. The routine for the visits was evolved through a 
careful pattern of training and adaptation, and normally involved 
discussions with one or more senior persons in each program, as well 
as some of their subordinates. Visits were followed up by mail and 
telephone where necessary. 

Site data were analyzed in the Palo Alto offices of the American 
Institutes for Research, and final decisions were made about the in- 
clusion of each program visited in the set of descriptions which forms 
Part II of the Final Report. Mo study was accepted for description 
unless data available indicated that pupils in the program had achieved 
statistically significantly better scores on standardized tests than 
had controls, or than national normative figures. 

In the analysis of site data it became evident that few if any 
compensatory education programs are free from blemishes of sampling, 
design, testing, data recording, or interpretation. Many apparently 
successful programs could not meet the strict criteria established 
for this study. Some that did may have done so through the undetected 
biases in their data, rather than by their educational significance or 
success. Inclusion of a program description in this report does not 
guarantee that its results are better than some not described, whether 
visited or not. 
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Part II of this Final Report comprises the descriptions of 21 
programs which met the study’s criteria. Each description was written 
according to a specially developed format, to provide a concise yet 
readable account of the nature, operation, and results of each program 
for principals, superintendents, and other interested persons. The 
U.S. Office of Education, sponsors of the study, requested that the 
descriptions be written so that enough detail was offered for a pre- 
liminary decision to be made in a school district about the desirability 
of attempting a locally modified replication. The descriptions include 
seven referring to preschool programs; 14, to elementary pupils, and 
six at the high school level. They range from small-scale experiments 
for less than 100 children to major programs involving many thousands. 
Some operate during school hours, others after school. Most are inner- 
city projects for Negroes or Spanish-speaking Americans, but some 
serve Appalachian white and other minority groups. The preschool 
programs claim improvements in intelligence ratings, generally speaking, 
while the others show benefits in terms of reading grade-equivalents. 

Each is unique in the treatment it provided. 

> 

Part I of this Final Report, although much more technical, is an 
intrinsic part of the study and should not be neglected. Apart from 
providing background to the study, and an account of its limits, it 
contains full details of the methods or procedures followed and of 
problems encountered. The typical characteristics of the programs 
described are discussed, followed by a tentative analysis of the 
programs’ components. The question of apparently well designed pro- 
grams which yield unimportant or no measured benefits of cognitive 
achievement is explored. Guidelines for program design and evaluation 
are proposed. A closing section deals with possible approaches to 
cost-effectiveness analysis for compensatory education programs. A 
list of references is followed by two extensive bibliographies, one 
providing general sources on the culturally disadvantaged, the other 
citing all materials collected during the study and relating to programs 
not described. 

In the tentative analysis of programs described, several common com- 
ponents were identified, such as a pupil-to-adult ratio of no more than 
seven to one. It is possible that these are c mponents critical to the 
success of programs in yielding measured benefits of cognitive achieve- 
ment, but no firm conclusions could be drawn on this point, as the study 
did not include a comparison between common components of (successful) 
programs described and ones (not successful) studied but not described. 

The guidelines offered for program design and evaluation concentrate 
upon a detailed, comprehensive, and accurate approach using appropriate 
statistical tools. 

The discussion of cost-effectiveness concludes that there needs to 
be a greater number of successful programs to choose among, as well as 
more comprehensive data on inputs and outputs. 
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INTRODUCTION 



Background 



Educational programs for children in poverty areas have become 
part of the fabric of modem America* They represent a determined and 
widespread series of attempts by political leaders, educators, and 
others to offset the shortcomings of the environments in which the poor 
of America live, by providing additional educational facilities. These 
facilities have been of many kinds, from summer school programs to pre- 
school programs, from new libraries to field trips. 

Twenty years ago probably few, besides political representatives 
from depressed areas and educators serving them, realized the importance 
to the nation’s life of these programs. Today, economists, political 
scientists, psychologists, physicians, businessmen, industrialists, 
and others agree that a heavy burden is carried by American society 
as a whole while such disparity remains between the results of educating 
the poor minority. This general concern has been reflected in the 
legislatures, where bills have been passed which provide federal and 
state funds for programs designed to improve the educational standing 
of children who come from culturally different, poverty-stricken districts. 

Negroes, ^perhaps by virtue of their presence in inner-city environ- 
ments, have been the subjects and beneficiaries of more educational pro- 
grams than have migrant workers, poor families of Spanish-speaking back- 
ground, American Indians, and Southern rural or mountain whites. But 
from all these groups, children come to school disadvantaged to the 
extent that their culture has not prepared them for school because it 
has not provided the experiences normal for the children the schools 
have been accustomed to teaching. 

When such children enter school, the cultural barrier between them 
and the teacher must be broken down. There have been several ways in 
which this has been attempted. Some programs have set out to train the 
teacher better (Jablonsky, 1966). The curriculum for such training 
courses has included the study of behavioral repertoires of children 
from Negro, Puerto Rican or Mexican cultures, in order to establish 
learning needs and strategies for preschool children. Examples of 
such studies are the Early Childhood Project (Deutsch, in New York City 
and the Tucson Cooperative Project (Henderson and Wetzel, 1968). Teachers 
have also studied the few available sets of materials developed specific- 
ally for use by particular groups of disadvantaged children. 
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Other programs have sought to provide early training for the 
pupil, so that he will understand better the new cultural milieu of the 
school. The Perry Preschool Project is an early example of this type 
(Weikart, Kamii and Radin, 1964). Still other programs have tried to 
improve the culturally different child's chances of learning in the school 
by providing supporting services during an extended school day. The extra 
time is used for remedial work, particularly in reading which is one of 
the chief tools of cultural transmission. The Homework Helper Program is 
an example (Peering, 1967; Cloward, 1967). 

/ 

The vast majority of programs, however, seek to hasten the accultur- 
ation process, often without due regard for the values of the cultures 
from which the pupils are drawn, and sometimes without sufficient 
attention being paid to the exact benefits of cognitive achievement to 
be derived. The result may be a sense of anomie among those so hurriedly 
"detribalized," of else profound dissatisfaction among those who find 

their acculturation is insufficient to guarantee good opportunities 
in Western society. Programs which succeed both politically and education- 
ally must be those which are based in the culture of the people they 
serve and which have high goals of cognitive achievement. 

. Nobody would pretend that it is not difficult to develop programs 
with such double goals. This study sets out to establish examples of 
programs which have at least served the second goal, that of cognitive 
achievement . 

Many of the programs of the past ten years have now been documented, 
some in books (for example. Bloom, Davis, and Hess, 1965; Hess and Bear, 
1967; Gordon and Wilkerson, 1966), others in proceedings of conferences 
(see Kvaraceus and others, 1965; U.S. Office of Education, 1963), still 
others in journal papers (Wilkerson, 1965) . A survey of this literature 
reveals that much of it is scarcely useful at all for evaluation purposes. 
Probably on account of the emotional undertones connected with poverty- 
stricken minorities, especially the once— enslaved Negro group, many of 
the writings cannot be judged to be objective; instead, they tend towards 
the diatribe or the proselytizing address. The better ones are heavily 
descriptive. Only the best report research results which are statistically 
sound and which might be used as a basis for future programs. 

Unfortunately, it seems true, that evaluation of programs funded by 
local school districts or by federal or state legislation has not been 
given high priority until very recently. In the midst of responding to 
the urge to do something about the problems which were painfully apparent, 
educational research administrators and others perhaps assumed that the 
success or failure of the programs would be self-evident. Bertrand 
Russell has claimed that educational research is always successful ; while 
we may disagree with him, there have always been good reasons to be 
quoted for not attempting too much evaluation. "We have been dealing 
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with h u ma ns ," we have said, "end they bring too aany variables to the • 
research. ** The ays tea of reinforcement which cam into operation as 
soon as sizeable funds becaae available from gov e r nm ent and other 
sources was one -which had not depended upon quantitative or concise 
evalu at ion nor upon measured evidence of vital changes in learners* 
behavior. It has been sufficient in many areas to prepare reports 
•bowing that pupils , teachers , principals, superintendents, parents 
and d i s tin g ui s h ed visitors liked the program, believed it to be success- 
fai, end thought it should be replicated. The reports in turn have 
beam used by mm senior administrators as support for their requests 
for further fu nding , regardless of die fact that the reports could 
• seldom show concrete evidence diet the most important parts of t he 
acculturation process had In deed been acc e l e rated. Only such acceleration 
will enable the pupil to shad his disadvantage and take hold of his op- 
portunities in our industrialised society, where gross cultural differences 
arm not easily tolerated. Because political pressures have ensured further 
supplies of money, nsny administrators have succeeded in their requests 
for extens i o n s of their programs. Thus not only their fund-seeking be- 
havior has been reinforced, but also the inadequate procedures for 
evaluating the effectiveness of the programs themselves, farther abetting 
this process has be en the scheduling deadlines for funding opperatlons 
idilch, in many cases, have necessitated that proposals for program 
renewal be submitted prior to the availability of data on original pro- 
gram success. 

-The educational world in general has tended to s u pp or t. the system 
too. the socialising purposes of American education, of vital importance 
in this century, have been harnessed to the newly apparent task of 
d ealing with the problems of disadvantaged children. Many educators 
have claimed that the solution to these probleaw lies in socialization, 
in persua din g meabers of poverty groups to become sound citizens and 
pursuers of the ronmnn good. Consequently, affective objectives 
feature prominently in the programs. The pro g r ams * success in these 
areas is usually measured by relatively unsophisticated questionnaires 
to faculty and students; in the questionnaires the questioner's attitude 
and role of authority is often only too clear, and a positively skewed 
distribution of responses is ubiquitous and practically inevitable. 

"Good" programs in the nation are identified by aany in the educational 
world as those in which some hypothetical happiness Index has been 
raised, however temporarily, and the impact of reality has been post- 
poned by Introducing an atmosphere of sweetness and light into the class- 
room, if not into the ghetto. 

Sometimes confused with those who seek bland euphoria for dis- 
advantaged children are those who seek motivation. The confusion Is 
not surprising, since the methods suggested are often strikingly 
similar. "Motivation" is the touchstone for success. If pupils can be 
persuaded to want to work in school, all will be well. Mobody would 
suggest that motivation is not vital, but the wort: set for the motivated 
pupil is equally vital. If he is motivated only to become verbally 
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fluent (in whatever dialect) or to "participate" in class, the most 
vital parts of the acculturation process are being neglected, at 
immeasurable cost in the future to the pupil and to our society. Unless 
■otivation for specific task accomplishment , e.g. , learning to read, 
to solve problems, to study art, to develop scientific thinking, can 
be enhanced, there is no assurance that the disadvantaged child will be 
more able to succeed once he is beyond the protective pale of the school. 

Concurrent with the recent demand for new programs yielding benefits 
of cognitive achievement, there has been a growing determination to 
attempt to devise analyses of existing programs. Planners -would like 
to identify the components of successful programs, then proceed to 
predictions of success and its cost and eventually to the design of 
hew programs containing components with the most favorable cost-effective- 
ness ratios. Since 1960, and especially since the President's announce- 
ments of August 1965, the spread of Planning-Progr wrnnl ng-Budgetlng Systems 
(PPBS) and related methods of analysis has brought a new approach to much 
of public accounting in the United States. Hitch and McKean's (1960) 
classic. The Economics of Defense in the Nuclear Age indicated the scope 
of such techniques in defense. Williams' monograph (1966) discusses their 
application to university planning, while Hatry and Cotton (1967) en- 
courage PPBS at the state and local level. 

The realization that such analysis nay after all be feasible has come 
at the sane time as certain crises in America's history. While America 
remains. the strongest nation economically on earth, her resources are the 
subject of competition by rival Interest groups. There have been civil 
disturbances at home, where the social conflicts are increasing. This, 
and the exercise of a very strong political and military influence abroad, 
have caused Interest groups to question closely the expenditures of federal 
money by their rivals. The defense "lobby” thus questions economic oppor- 
tunity expenditure or economic aid to non-aligned developing countries. 

Education, including the education of the disadvantaged, has not 
escaped the. general demand that expenditures be justified by hard evidence 
rather than opinion. The Office of Education has shown itself to be fully 
aware of this demand, and stat scents by top officials over the past year 
or more have particularly stressed the need for evaluation of programs. 

The techniques of cost-benefit analysis and cost-effectiveness 
analysis have seldom been applied in education, although these and other 
techniques were discussed at a recent Washington symposium, entitled 
Operations Analysis in Education. Dr. Gerhard Colm, a member of the Ad- 
visory Group to the State-Level Finances Project of the George Washington 
University, has pointed out elsewhere that: 

In the military field, cost-effectiveness studies have been 

most useful when .... a single clearly defined objective .... 

could be pursued by alternative methods. In many civilian 
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programs (including education), we cannot assume that one 
single goal is -given — the same program often serves 

several objectives techniques for determining 

optimal solutions for programs with multiple objectives have 
yet to be developed. The problem is further complicated by 
the present difficulties in obtaining q«iantitative measures 
of many important economic and social impacts of programs, 
even if they are comceptually clearly defined. 

At the present time, no tested methodology is available ....* 

The field of cost— effectiveness analysis in education is scarcely 
ex plo red, as filldavsky (1966) has noted. One of the first realistic 
incursions was made in the study of Technomics Inc. of the feasibility 
of cost-effectiveness analysis for Title I of the Elementary and Second- 
ary Education Act, prepared for the Division of Operations Analysis, 
Rational Center for Educational Statistics, U.S. Office of Education 
(Technomics, 1967). The conclusions of these and other studies are 
discussed later in this F inal Report, but nay be summarized here. In 
short, cost-effectiveness analysis in education is premature until the 
inputs and outputs of the schools are more easily identifiable and more 
comprehensively measured. 

This study early encountered similar problems as those which led 
to the conclus ion that cost-effectiveness analysis in education is pre- 
mature, and the decision was taken to pursue instead the objectives of 
describing and analyzing the components of programs which had yielded 
benefits of cognitive achievement. The descriptions could serve the 
purpose of providihg examples of programs worthy of replication, with 
local modifications, while the analysis of the components might provide 
a. basis for the eventual development of realistic and cost-effective 
models for future programs. To serve these ends, it was necessary to 
identify and select those programs which met certain strict criteria. 



Limits of this Study 

These criteria can be explained best by wane of a glossary. At 
the same time the limits set by the U.S. Office of Education should be 
established, distinguishing them from those set by the American 
Institutes for Research. The Office of Education asked that the study 
include educational programs for disadvantaged children. "Educational 
programs" was interpreted widely to encompass not only those providing 
formal instruction in schools but also those seeking out children in 
their homes or on the streets. "Disadvantaged children" was taken 
to be a generic term, however inaccurate or disliked, which included 



* In his Preface to Hatry and Cotton (1966). 





children of Negroes, Mexican-Americans (both migrant and resident), 

Puerto Ricans, American Indians, and Southern rural and Appalachian 
Whites. 

The Office of Education further asked that the study include pro- 
grams serving children from preschool to grade 12. This was interpreted 
widely to include the Infant Education Research Project in Washington, 
D.C., which began tutoring children from the age of 15 months. The final 
specification of the Office of Education was that each program show 
"measured benefits of cognitive achievement." The American Institutes 
for Research, on advice from the Office of Education, interpreted 
"■easured benefits" to mean gains, measured by standardized tests, 
significantly greater than pupils would have made had they not received 
the program. Normative progress was judged from control groups or norm- 
ative data. "Cognitive achievement" was at first taken in its widest 
sense, with the result that a few foreign language and science programs 
were included in the early stages. It was later decided to limit the 
study to programs aiming at increasing cognitive achievement in language 
or number of both. Standardized tests in these fields were judged by 
the American Institutes for Research to include intelligence tests, 
since these were in the opinion of the A.I.R. research staff more measures 
of achievement than of innate ability in the context of the programs. 

The American Institutes for Research decided to include only those 
studies reported later than 1962, since earlier ones would be too far 
removed from present-day conditions. 



Related Research 



It is inappropriate to include under this heading the many documents 
on which this report is based, as these are referred to in other places 
in the body of the report itself. Rather, related studies and surveys 
of compensatory education in the United States should be quoted here, to 
indicate their relationship to this study. 

Compendia of compensatory programs have appeared from time to time, 
ranging from an early booklet prepared by the Office of Education (U.S. 
Department of Health, Education, and Welfare, 1963) to volumes such as 
those of Hess (1965) , Gordon and Wilkerson (1966) , and Pines (1967) . Other 
surveys have been published as journal articles (see Wilkerson, 1965). 

While many of these contain descriptions of programs, the descriptions 
were not prepared as part of a deliberate evaluation, consequently they 
contain much qualitative and impressionistic material not suited to the 
purposes of this study, in which quantitative data were the basic criteria 
for selecting programs for description. The compendia were indeed helpful 
in establishing initial lists of programs to be investigated, as well as 
providing secondary sources for the description writers in this study. 



Several sources of national information were also examined. The 
Coleman Reporc (1966) on equality of educational opportunity helped the 
A.I.R. research team to see the broad picture of education for disadvantaged 
children across the nation, as well as pointing out some sophisticated 
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(if sometimes mistaken) statistical techniques for the analysis of data. 

A report by Peerboom (1967) on pre— Title I compensatory education 
prograw was made available to A.l.R. by the Division of Compensatory 
Education of the Bureau of Elementary and Secondary Education in the 
U.S. Office of Education, and highlighted a number of programs for 
consideration. Also, the National Advisory Council on the Education 
of Disadvantaged Children has published a series of reports on compen- 
satory education nationally (e.g. , National Advisory Council, 1967). 

The Council provided a complete set of these reports for this study, 
and general trends were identified from them as background. 

The Division of Compensatory Education, referred to above, has 
und ertaken several in-house surveys, the reports of which were also 
made available to A.l.R. The basis of evaluation used in these surveys 
was different from that of this study, since the surveys did not attempt 
in-depth analysis of selected programs. Similarly, other in-house 
studies made of compensatory education by the Office of Program Planning 
and Evaluation of the Office of Education were of only indirect relevance, 
but did provide valuable background. 

The single source which provided most direct help during the course 
of this study was probably the Technomics (1967) report. While the ap- 
proach and techniques used in the Technomics project could not be trans- 
ferred in toto to this study, the A.l.R. staff was able to gain much from 
Technomics experience. 
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METHODS AND PROCEDURES 



Selection of Programs to Visit 

Literature search . The initial source of information on programs 
for educating disadvantaged children was an extensive search of the 
literature. The primary purpose of this activity was to develop a 
large pool of. educational programs from which subsequent selections 
could then be made for in-depth study. Information collected during 
this stage included basic program identification, description of the 
pupils involved, the behaviors measured and benefits claimed, and the 
names and addresses of appropriate contact personnel. The form 
employed for recording this information is included as Appendix A. 

In order to facilitate the selection of programs to be visited, a 
two-way rating system was developed. Programs were classified accord- 
ing to whether they included controls and apparently achieved measured 
benefits; whether they apparently achieved benefits but were lacking 
in controls; and, finally, whether the study involved controls but 
without measured benefits. Secondly, on the basis of a quick review 
of the project report or its summary, a design rating form was completed 
(see Appendix B) . This form was developed during the study as an instru- 
ment for gross evaluation of program quality in terms of whether the 
report provided information on such essential items as population char- 
acteristics, sampling procedures, program objectives, criteria involved, 
measuring instruments 'employed, and completeness of the evaluation 
findings. 

The Educational Resources Information Center (ERIC) of the U. S. 
Office of Education provided an excellent source of information on 
documents, particularly concerning earlier programs. Probably the single 
most comprehensive index of documents on educating the disadvantaged is 
the ERIC report Catalog of Selected Documents on the Disadvantaged 
(1966). Other USOE compilations of project and proposal information, 
such as Pacesetters - Innovation in Education (USOE, 1966) and Current 
Project Information (USOE, 1967) served as valuable additional sources. 

In addition, the monthly copies of the ERIC journal Research in Educa- 
tion (USOE, 1966-68) were carefully screened for leads on relevant 
educational programs. 

Arrangements were made with the ERIC Clearinghouse on Educational 
Media and Technology, located at Stanford University, to review the 
microfiche copies of the reports which had been identified as worthy 
of further study based on the information contained in the USOE index 
publications. Two staff members of AIR spent approximately 6 weeks 
reviewing microfiche and other materials at this ERIC Clearinghouse. 
Sometimes, it was possible to accept or reject an educational program 
based upon a review of the abstract; however, in many cases a more 
complete study of the report was necessary. The criteria established 
for the selection of educational programs at this stage were as follows: 
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a) Program report completion date of 1963 or later 

b) Directed toward disadvantaged children at the elementary 
or secondary level (preschool to grade 12) 

c) Displayed gains in cognitive achievement, with the primary 
focus upon reading or language arts, mathematics, or IQ 

d) Showed evidence of good experimental design sufficient to 
justify some degree of confidence in the findings 

Bibliographic information was also obtained from a number of other 
sources. At the request of A.I.R. the Science Information Exchange (SIE) 
of the Smithsonian Institution prepared a list of references to relevant 
educational research projects in progress. Also of particular help 
were the references contained in the bulletins of the ERIC Information 
Retrieval Center on the Disadvantaged; the references to educational 
programs contained in Gordon and Wilkerson (1966) ; and the Hess Inventory 
of Compensatory Education Projects (1965). 

Whenever documents could not be located through the local ERIC 
Clearinghouse, the facilities of the Stanford University Cubberley and 
Main Libraries were used. If this failed, copies of relevant documents 
were ordered by mail. Mail requests were also sent to school districts 
and other agencies to obtain up-to-date information on programs where 
the only available information was contained in abstract form, in a 
proposal document, or in very early progress reports. All told, some 
300 such letters were sent out. 

It is estimated that approximately 2000 documents were screened 
during the literature search phase. From this phase a pool of programs 
was derived, 185 of which were selected because they appeared to meet 
all or most of the criteria established for inclusion in the present 
project, as listed previously. For each program selected at this stage, 
three cards were completed: a bibliography card, a project description 
card, and the design rating form described earlier in this section. Of 
these 185 programs, 93 were tentatively designated as worthy of further 
study while the remaining 92 were considered as reserve or back-up 
programs. 

Conference of experts . Project funds and the time available did 
not permit site visits to all of the programs that had been identified 
during the literature search stage. This condition had been anticipated 
during the early planning of the present project and accordingly prepar- 
ations had begun, even before the start of the project, to assemble in 
Palo Alto a panel of nationally recognized experts to provide assistance 
in final program selection. A two and a half day conference was convened 
six weeks after the project was initiated. In attendance were 10 
national experts, three officials of the U. S. Office of Education, and the 
A.I.R. staff. A list of consultants attending the conference is pre- 
sented as Appendix C. 
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While the conference provided valuable guidance concerning a j 

variety of problem areas, including program categorization, component 
and benefit identification and categorization, and data analysis, its 
primary purpose was to assist A.I.R. in the selection of educational 
programs for in-depth study. As a result of the conference review and 
discussion of the individual project cards, 80 programs were selected 
as definitely worthy of in-depth study. The remaining programs were 
placed on a reserve list. 

The resulting list of programs to be visited was then submitted to 
the U. S. Office of Education, to the Carnegie Corporation, and to the 
National Advisory Council on the Education of Disadvantaged Children. I 

At the same time, the literature survey was being continued on. a reduced 
basis in order to exploit all available sources of information on new 
educational programs. Based upon the suggestions received and the j 

new program references identified, detailed preparations for the site 
visits were begun. 

Site Visiting 

Site visit preparation . Because of the need to obtain a great j 

deal of specific program information in a short period of time, it 
was necessary to develop a site visit routine which would ensure that 
all necessary information was obtained with a minimum burden upon pro- 
gram personnel. It was also deemed essential to avoid unnecessary 
structuring of the interview and the consequent stifling of promising 
discussion areas. 

As a first step in developing site visit procedures, the Project 
Director and the Associate Director made preliminary visits to several 
programs. While a draft checklist had been developed on the basis 
of past experiences and the results of previous studies (Technomics, 

1967), the interviewing at this stage was only minimally structured. 

The primary objective was to obtain information on the effectiveness 
of alternate site visiting strategies, leading ultimately to the devel- 
opment of a balanced approach, i.e. , neither completely unstructured 
nor unnecessarily rigid, which could then be applied by all site visitors. ' 

As a result of the experience gained during these visits, the 
initial site visit checklist was revised. The checklist, in fact, con- 
tinued to undergo minor modifications throughout most of the data 
gathering phase. Some 100 separate items of program-related information 
were solicited under the following headings: 

a) Basic Data 

b) Neighborhood Charaj-oristics 

c) School District Characteristics 

d) Pupil Characteristics 
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e) Personnel Characteristics 

f) Treatment Characteristics 

g) School (Personnel) 

h) School (Provision of Services) 

i) Costs 

Site visit execution . The site visits were conducted by five 
senior staff members, with the assistance on two occasions from other 
staff. In preparation for a specific site visit, all the relevant 
literature available to A.I.R. was reviewed and compared against the site 
visit checklist of required information. By this process, it was 
possible to determine those areas where adequate information was already 
available and those areas where the site visitor should concentrate 
on obtaining first hand information. In this way, the time of both 
the program personnel and of the site visitor could be conserved. 

The site visitor was provided with a file containing all available 
reports on the program, any correspondence with persons in that city, 
and a marked-up copy of the site visit checklist. This material was 
studied en route to the program site. 

Site visits were scheduled to those locations where information 
could be obtained on one or more programs selected as definitely worthy 
of in-depth study. In addition to having background materials on these 
high priority programs, however, the site visitor was supplied with 
the available materials on all of the compensatory educational programs 
that had been identified with that particular location. In this way, 
he was able to obtain first hand information on those programs that 
had been tentatively rejected or placed on the reserve list during 
the literature search phase of the present project. This served as an 
excellent cross check at very slight additional cost to the project. 

Also during the interviews with school district evaluators and compen- 
satory education coordinators, the site visitor was able to identify 
any other eligible programs which had not been identified up to that 

time. 

The site visitors had appointments arranged with the senior persons 
available for each program, and, depending on the program, with one or 
more of their subordinates. A particular effort was made to meet first 
with the head of the evaluation team; next, a meeting was often arranged 
with the coordinator of specially funded programs. Finally, the program 
was discussed with those who had administered or actually operated it. 

In general, the site visitors did not visit the classrooms of programs 
since in most cases the classes had been dismissed for the summer. This 
was no handicap to the survey, since the most efficient use of the time 
available was to consult those who were directing or implementing the 
program. When new project documents were received during the early part 
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of a site visit, these were examined carefully before detailed question- 
ing of the program director. 

During the site visits, information was solicited concerning each 
of the items marked on the checklist. The site visitors also became 
adept at following slight clues which led to the discovery of signifi- 
cant facts about the program in operation, or about the local context, 
which would not normally be reported. They quickly learned to extend 
their questioning in such cases, and to request interviews with other 
personnel mentioned. 

The site visit follow-up procedure comprised several parts. First, 
the site visitor usually tape-recorded his full impression*: > as well as 
amplifying his written notes from the interview, at the first oppor- 
tunity. This was often in the evening at a local hotel. Second, the 
rough transcription of a site visitor's record, together with the pro- 
gram file and any new printed materials obtained during the site visit, 
was studied by one of the staff in Palo Alto, and a draft was prepared 
according to a particular format. If there were gaps in the draft on 
account of information being missing, questions requesting these data 
were drafted. The site visitor then became responsible for letter or 
telephone follow-up to secure the missing details. The site visitor 
was also responsible for sending appropriate letters of thanks to 
personnel who had assisted in the survey. 

Problems in the Analysis of Data 

Components and factors which hindered analysis of their effects . It 
may seem superfluous to define in this study our concept of a component, 
except to the few who are familiar with the controversies sparked off by 
the introduction of the mathematical-statistical techniques of "factor 
analysis." Here "component" is not merely analogous to the "factors"; 
it is a close parallel, and in special circumstances could even become 
synonymous with it. 

It is any unitary system of action, pattern of events, collection 
of artifacts, or combination of these, which may be introduced into 
the educational system without variations from one point to another; 
there may be many elements which can be recognized, but if internal 
relationships are invariant the complete system is the component; if 
relationships do change with the occasion, the system at once reduces 
to a combination of several components. Thus if a library is always 
staffed, the component is made up of the elements books and staff; but 
if some collections of books are not supervised or controlled by a 
librarian, "librarians" and "book collections" are two components. 

Even for those who find this explanation necessary, this may sound 
excessively legalistic and verbose, especially when it is known that this 
study almost certainly though inadvertently has failed to recognize 
some "components" as collections of elements, and others as mere elements 
in a larger but undefined component. Yet it should throw some light on 
some of the problems which were faced. 
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Little or nothing hindered identification of intended components; 
they were for the most part, and almost invariably shaped bricks, 
neatly labeled, clearly identified, and consciously, deliberately, and 
methodically used to build the programs from the start; carefully 
defined concepts included in the designs from the beginning. 

Once there, nothing encouraged, or even permitted analysis of their 
effects, for as is well known, it is possible to find particular values 
for a number of unknowns only when at least as many equations can be 
found (all of them different) to link them. In other words variations 
due to single causes can be identified only in the context of variability 
in application. If for every class one reduces class size to 25, imports 
teacher aides, gives remedial training by specialists, provides a 
library with librarian, organizes excursions, buys slide-, strip-, cine-, 
and overhead projectors, visits the homes, and appoints a clinical 
psychologist to the staff, it would require quantitatively unique com- 
binations of the above to be able to determine the part played by the 
psychologist — if a mathematical solution were possible. Since this 
study deals with probabilities and not certainties and must use statis- 
tical rather than mathematical solutions, scores if not hundreds of 
schools would be needed, even without the additional complexities of 
variations in age, sex, curricula, size of school, ethnic groupings, and 
socio-economic status. 

This deliberate identification and inclusion of components yet 
with cavalier treatment of the demands of analytic studies may be 
poor research design — but it is surely sound educational practice. 
Compensatory education as seen in this study was an onslaught upon the 
cumulative effects of ages of poverty, both financial and cultural. 

An onslaught is a battle, and in battle there is freedom to select 
targets; but a crucial battlefield would not normally be divided into a 
Latin Square to be able to test selectively the effects of several new 
armaments or ammunition! In an analogous situation, who would tolerate, 
let alone expect, unpredictable experimentation from a medical prac- 
titioner? ^f fault there is, it originates with those who provided the 
enormous funds to blanket the nation's education departments; which did 
they think they were buying — action, or research? Or was there 
another motive? 

Action and research are to some extent incompatible. The first 
seeks to guarantee a predetermined outcome; axiomatically it spares no 
effort and is entirely dependent upon the existing store of knowledge 
and information; time is of the essence. Research, on the other hand, 
is often slow; and unless it deliberately and selectively restricts the 
scope of action it may seriously handicap the attempt to add new know- 
ledge to the existing store. 

Be that as it may, Oakland and Cincinnati Boards of Education are 
outstanding examples of institutions which mounted huge programs built 
around substantial numbers of defined components, with little or no 
chance of evaluating their independent contributions. Both made 
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imaginative, energetic, and purposeful use of the funds they had been 
given, budgeting carefully. Both tried to incorporate as much practice 
adjudged compatible with the present state of pedagogical theory as 
possible, together with innovations for which a priori cases could be 
made. Each had made full use of the full range of experts from language 
specialists to programmers of computers, and had made provision for Lbe 
only kind of research possible under the circumstances: that of deter- 
mining whether the combined effect of all their efforts and components 
had produced measureable if not significant improvements in their pupils. 
They ran remarkably parallel courses and their findings were similar. 
These might be 3aid to be at least as important as anything described 
in this report. 

However, the attempt to feed a balanced educational diet was not 
the only obstacle to the determination of the relative Importances of 
the elements, although it must be obvious that no greater obstacle to 
piecemeal study can exist than thorough and consistent "homogenization" 
of the ingredients. There are concomitants. 

When the achievement of the goal is what matters it is obviously 
necessary to depend heavily upon the wisdom and insights of those who 
are most directly responsible for action, and to allow them the flexi- 
bility in selection of schools, of classes, of pupils, of extent of use 
of components, and , for that matter, even of selection of components or 
introduction of unique ones. These freedoms are vital, but would throw 
heavy burdens on a research design which at least recorded such varia- 
tions; but such recording presupposes measurement, which in turn 
requires time, finance and personnel, reducing the resources available 
for the achievement of results. Small wonder that one such program 
(admitiwily not one of the larger ones) had allowed a budget of about 
$400 for testing! In this same program its director mentioned that one 
of the most important duties and functions was that of becoming aware 
of significant differences in interpretations and implementation, so 
as to be able to coordinate the programs as well as to be able to 
interpret the results. 

There is one other line of argument which must have influenced 
senior educational policy makers. Given the present climate and sense 
of urgency, underscored by the very magnitude of the funds which had 
become available, no one can be blamed for seeking results rather than 
knowledge, and success before economy. More than that, it is a tenable 
argument that both research and economy can follow upon success, and 
economy at least upon failure; if even a shotgun hits nothing why shoot? 
Otherwise get your bird first and then Improve marksmanship. In one 
way this is even in itself an economy, since any success immediately 
reduces the target area. This line of reasoning conceals at least two 
flaws; the analogy of the shotgun breaks down, for pellets do not cancel 
one another out or otherwise interfere with one another, nor is there 
such a thing as a failure to bring down a bird because too many pellets 
hit it. But it is conceivable that one educational component could 
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militate aginst the effects of another, and it is believable that too 
much of a good thing can be harmful; that excursions can reduce time 
available for reading; that concentration on reading can handicap the 
teaching of arithmetic; that film shows can so swamp a child with new 
facts that he is overwhelmed; that too small a ratio of pupils to 
teaching staff can result in enough concentrated attention to demands, 
to produce rebellion; and that transporting backward pupils into a pre- 
viously successful and fast-moving class in another school, can so slow 
the pace of the original pupils to the point of tedium, that it more 
than offsets the gains to the newcomers. No doubt such risks as there 
were, were well-recognized, guarded against where possible; danger 
signals were watched for. But maximum gains are always closely associ- 
ated with maximum risks, and the only reasonable alternative to the 
rejection of funds is to allow flexibility in their use. 

Both Oakland and Cincinnati applied the well known "black box" 
principle of research; that is to say, given a situation in which pro- 
cessing is too extensive and complex to allow of systematic or logical 
attention, it is always possible to select points of input and output 
which do permit observation, measurement; and comparison. Thus for 
example standardized tests of reading ability with known norms can be 
applied to groups of pupils fairly homogeneous in terms of age, grade- 
and f socio-economic status, before the tumult and again after a year of 
it. Although in practice there are substantial problems involved, in 
theory comparisons of the before-and-after results, as well as of each 
with corresponding norms of total populations including the "over- 
privileged" and "under-handicapped", it should be possible to form an 
opinion of the overall value of the processing. On the whole it was. 
Many will have a far higher regard for the mountains of effort than for 
the mice to which they gave birth. The point is that even if it is 
accepted that "no price is too high to pay for success, however small", 
partitioning of insignificant gains is not a practical pursuit. Dr. 
Jacobs of Cincinnati concludes one of his reports with a summary of the 
findings and pertinent comment (Jacobs, 1968). 

He was dealing with The Elementary Remediation and Enrichment 
Project, the most extensive Title 1 effort to improve the education of 
Cincinnati's disadvantaged children, which began in February 1966, and 
continued with an even larger scope through the 1966-67 school year. 

He identifies the following components which among others had been 
important enrichment activities: 

Remedial instruction in fundamental areas 

Parent involvement 

Provision of learning resource centers 

Summer learning camp 

Jacobs adds: 



Evaluation of the project in the light of its functional 
objectives, such as improving self-image and classroom performance 
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yielded few positive results. Occasional small differences were 
noted favoring project pupils, e.g., classroom marks in reading. 

Similarly, findings from the evaluation of special instruc- 
tional activities generally showed no significant difference. 
Achievement data comparisons showed only one area of instruction 
that seemed to produce positive results. Remedial Language 

Arts instruction seemed to have raised achievement in spelling 
and language. 

Then he reports a common finding associated with little or no ob- 
jective and measurable cognitive gain — the subjective approval of 
all 'concerned. 



Although significant differences were not found even in 
the separate evaluation of three project components, assess- 
ment of pupil and teacher reaction produced strongly favor- 
able results... 

Money might not buy success, but let us at least fail more cheer- 

filly! Dr, Jacobs writes a "postscript" which deserves wide circula- 
tion: 







^ this project is having a desirable impact on target 
pupils, as reflected by pupil and project staff judgments, then 
we must either accept these judgments as intrinsic signs of 
success or discover new and refined ways of detecting differ- 
ences where they exist. Perhaps we are looking for change in 
the wrong places. Perhaps our instruments for measurement are 
too crude. Perhaps our evaluative techniques are inappropriate. 

And perhaps no changes are occurring in target pupils. We 
simply don't know. 

It is not coincidence that projects seem to fall naturally into two 
classes : 



1. Very large scale, involving many schools, large numbers of com- 
ponents, very substantial funding, with personnel qualifications cover- 
ing a wide range and involving a variety of specialists, well able to 
cover the needs of any research involved, yet with action rather than 
research as the goal. 



2. Small, restricted studies, perhaps one or two components, little 
or no funds except the dedication of a few persons who are striving to 
test limited hypotheses or, as often, to vindicate their dedication to 
a particular school of thought — people with axes to grind, or at 
least causes to serve. 



Several forces independently reinforce this tendency to a dichotomy. 
As we have said, research tends to be expensive, and the expenses 
increase geometrically as the complexity of the design and number of 
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variables increase arithmetically. Complex designs call for enormous 
samples; society can tolerate research upon itself, most especially in 
sensitive and vital areas, in small quantities only ~ small enough 

not to cause politically important repercussions, since politics thrives 
more on action than on wisdom. 

On the other hand, innovations occur to individuals not to com- 
mittees, and one or two at a time. All innovations are suspect until 
they are proved, and public funds are seldom given for anything that 
is not a gilt-edged security, The man with the idea must depend upon 
his own resources; he is likely to be a specialist in one field only, 
while modern research demands teams of specialists. 

Small wonder then that Boards of Education are well equipped to 
do research, report what research they do critically and objectively, 
making few mistakes and moderate claims, yet strive rather for results; 
and that small operators sometimes make enthusiastic claims based upon 
inadequate designs, often upon erroneous reasoning. 

. Problems of_ missing data. In this study it was therefore recog- 
nized that the emphasis was on the discovery of new and successful 
instructional techniques rather than of sophisticated demonstrations 
of success. In other words, it was hoped that the search would be 
simplified often by the authors whose descriptions were being read. It 
was hoped that their orderly and systematic development of their themes, 
with proper presentation of adequate summaries, data, and tests of the 
data would make matters simpler. But it was expected that sometimes 
there would be amateur statistics prepared by professional educators. 
This would not matter provided that enough evidence could be unearthed 
to support claims made. Facts could be hidden, but they had to be 
there. There are few canons and essential steps, and a wide variety of 
satisfactory designs; where the minimum could not be found explicitly 
stated, implicit indications were sought. 

In one form or another, the following features were essential to 
the assessment of the program: 

Hypothesis A description of the new element, or treatment, or 
method which was hoped to be shown superior to an alterna- 
tive and of the specific changes it was expected to produce, 
preferably with a linking rationale. 

Population — Those to whom the treatment could be applied or 
who could be expected to benefit from it. 

Sample The smaller, practicable number, chosen from the popula- 
tion for purposes of demonstration, and the way in which it 
was chosen. 
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Base — The reference point of departure or the way in which it 
was to be obtained; the beacon from which movement could be 
shown. 



Measurement — Both the instruments used to detect change, and 
suitable summaries of the values obtained had to be given. 

Comparison — There had to be some way of determining a difference 
from values derived from the base. 

Significance — It was essential to be able to distinguish reliable 
or meaningful results from fortuitous ones on a probabilistic 
footing. 

Importance — A more tendentious concept, but at least in the 
extreme case it is pointless to quote trivial gains merely 
because they can be shown to be real. 

This study encountered trouble under each of these headings sooner 
or later. Sometimes a letter or a phone call, or another visit produced 
cl fication, though more often than not information not already obtain- 
ed was, in fact, inobtainable. Once or twice respondents were nearly 
irritated by repeated requests. Since there is nothing to be gained by 
provoking acrimonious debate, the anonymity of the sources will be pre- 
served. Some of the points of criticism could undoubtedly be resolved 
to the satisfaction of the opposition, if not to the authors of this 
report. The principles involved, however, are important enough to 
expand on and to illustrate on occasion. There is no intent here to 
set up a rigid framework upon which to hang reporting, rather some of 
the causes for dismay should be displayed. 

Most of these causes could be reduced to a single one: The necessity 
to deduce from inadequate or misleading exposition . This might be put 
in another way; more errors of omission were encountered than of commission. 
Each of the essential features will serve to focus the discussion of 
what was lacking. 

Hypothesis . As mentioned elsewhere, description was seldom lacking 
of the components involved in the treatment to be applied; these might 
be "appointment of teacher aides" or "reduction in class size." It was 
not as universal to find a description or rationale of the mechanisms of 
change, or even of the changes that were expected. More often than not 
this did not matter because the intentions were obvious; for example, 
reduction in class size yields more individual attention, and if overall 
school performance is quoted, thdt is what it sought to change, and such 
change would be plausible without the need to dot the i’s or cross the 
t's. Mo example was found of a face issue of paperback novels linked 
with gains in arithmetical achievement; what were found were remarkably 
close parallels in which the investigator almost certainly deceived 
himself as to the hypothesis he was testing by failing to trace linkages 
and to examine the contents of the test he used; at the very least such 
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cases encourage others to misconstrue the demonstration. 

Population . Only infrequently was the population at which the 
demonstration was aimed properly circumscribed, and again because this 
Was usually clear enough from the description of the sample used. Too 
often, however, there were aspects of the sample which deserved a close 
scouting to see whether they were desirable parts of the intended des- 
cription of the population. Au example of this will be found in the 
next section. 

In some cases, several samples were used, each with unique features 
which made it difficult to decide just what population was intended, 
and raised doubts about the representativeness of any one of them. 
Dissimilarities between experimental and control samples increased the 
problem. 

Sample . The easiest sampling method to defend is the random one. 

It is also, especially in schools, the most impractical and most expen- 
sive; the most logical and practical in an educational setting is 
cluster sampling in which one draws complete school districts or parts 
of them, or schools, or at least classes. Strictly speaking, this 
alters the basis for statistical inference, but in a s cable system poses 
few other threats. However, radical changes are in progress in some 
schools, in some cases leading to increased heterogeneity or even to 
bimodal distributions on important clusters of variates. In such cases 
it becomes of considerable importance to determine whether gains have 
salient associations and this can be done only when the sample is fully 
described. For example, pre- and post tests of a class, half of whom 
have been "bussed in", may show gains which may or may not conceal 
losses in part of the class; but the findings are applicable only to a 
population with the same composition as the sample. The effect of an 
additional constraint may not always be known, but its description and 
emphasis may help to guard against unjustified extension of the results. 
In one study, voluntary participation by the pup\l was essential; there 
is no way of telling whether the gains obtained could also be expected 
from conscripts. 

In one case, the description clearly indicated that the treatment 
had been devised for handicapped children; this then was the population . 
As no control group was used, the most reasonable base would have been 
"below-normal progress of less than one grade per year", in the ability 
under study. Progress in the sample under treatment was well above 
normal — as it should have been since they turned out to be a group of 
"over-privileged" children with a large advantage in mean intelligence. 
As this was a very minor part of the study, it did no harm. 

More serious because uncheckable, were the cases where sample 
losses (sometimes, worse still, with replacement) occurred between pre- 
and posttesting, so that part at least of the gains claimed could have 
been due to systematic biases thus introduced. In such cases, only 



access to the original data can answer questions raised, in an extreme 
case, for example, a loss of the bottom ten percent can cause an 
upward shift of one-fifth of a standard deviation to the mean. 

Base. The base, explicit or implied, was a prolific source of 
trouble* Often there was an attempt to provide a control group} if 
looseness in procedure afflicted sampling, the same looseness appeared 
in drawing the control group making it possible to double the effects 
of biases. Sometimes the method of obtaining the experimental sample 
virtually guaranteed inappropriate control groups} for example, in one 
case so large an experimental sample was drawn from one school that the 
remaining pupils had to be supplemented from another school to make 
the control group large enough, making the constituency of this, at 
least to some extent, uncontrolled and problematical. 

Matching individuals on important variates to provide the control 
group is a method which has lost popularity over the years} particularly 
when matching is done simultaneously on-, two or three variates it can 
lead to prodigious losses of available subjects whose data can be put 
to good use with modern statistical techniques. Nevertheless it can be 
® knightly suit of armour with hardly a chink. One research team un- 
wittingly exposed such a chink for a thrust when it used IQ as a 
"variate" for matching purposes} IQ is not a single variate, but a 
combination of at least two important ones (capacity and age) arranged 
in such a way that a deficiency in one of them can be compensated for 
and concealed by a low value in the other, making it theoretically 
possible to have corapletely dissimilar experimental and control groups. 

A fallacy with wide acceptance, and which marred a significant 
proportion of studies, some of them large ones with considerable finan- 
cial backing, is that modern statistical techniques (notably analysis 
of covariance) can compensate for radical differences between experimental 
and control groups and even for different tests of the same general de- 
scription. This is no place to deal with such complex considerations in 
any detail though some of the problems of, and objections to the pro- 
cedure will be touched on later. Suffice it to say that some research 
workers took unwarranted liberties with these techniques and permitted 
undue latitude in selection of control groups. 

Failure to make specific provision for a base was not necessarily 
an insurmountable handicap as long as the composition of the sample was 
clear and measurements given were normative} elsewhere will be found a 
neat^ graphical approach to the problem of finding suitable grade norms 
for disadvantaged" children against which to check progress. However, 
scores were occasionally given in a form which did not allow of such 
comparisons, and the mere demonstration of gain could only be tested 
against a meaningless "null hypothesis" of "no gain." 

Measurement . Of all elements in the research process, the one 
with the greatest and most pervasive influence, the one most subject 
to misuse, abuse, or simple misinterpretation is probably that of measurc- 
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ment. This is perhaps true in any field, but more so in the behavioral 
sciences since there, ratio measures are almost completely absent, and 
the measures that are used are more dependent upon arbitrary and in- 
constant choices, with less control and more confusion of the dimensions 
involved. It was rare to find a project in which the measures used did 
not give rise to some suspicion. Typical of the problems encountered 
ygg the "changing of horses in midstream"} the use of one publisher s 
test for a pretest because it was designed for non-reading beginners, 
another's for a posttest because it measured reading achievement; 
or the use of test A at one level followed by test B from the same 
stable at a higher level. If the aims had been to establish predictive 
systems, this would have been logical; but not when the goal is to 
measure gains on a single continuum. The point is that the title of 
a test is a rough guide only to its content and number of dimensions 
involved. Not only is "Reading Readiness” not synonymous with "Reading 
Achievement", but even two reading achievement tests may differ in the 
influences which can produce change. In such a situation there is a 
strong temptation to which most succumb , to try to compensate for 
crudity of experimental designs by using highly refined statistical 
processes, but that is another story. 

Even when a study was neat and tidy, and efficient in its choice 
gnH use of tests, often in this study the statistics needed could not 
be found; or else they were suprisingly selected; or on occasion, gross 
arithmetical errors were suspected and found, or strong evidence that 
these must have occurred. Why in this day of desk calculators if not 
computers, did so many depend upon the less efficient median instead of 
the mean? And having done so made no attempt to test the significance 
of the median gains claimed even though the standard error of the median 
is to be found in elementary textbooks? Worse still (even when means 
had been calculated sometimes), many did not even give standard deviations; 
understandably hardly anyone quotes original data, which left for this 
study only devious expedients and approximations in a few cases, and 
no hope of a fair assessment for a larger number. In at least one case 
such gross arithmetical errors had been made in simple subtraction of 
means that a "significant" gain disappeared when recalculated. In 
another, significances of gains had been tabulated, with almost no 
supporting data; it was possible to show that one of the missing items, 
correlation of pre- and posttests , separated by an interval of a year, 
must have been of the order of .93 — a surprisingly high figure. 

The most important single statistic (after the measure of central 
tendency used, mean or median) is the standard deviation, and this was 
missing far too often; even more often the standard error of the mean 
was absent which would have been only a little worrying if the SD had 
been available. Almost as important, especially when differences 
between gains for experimental and control groups were being compared, 
was correlation between pre- and posttests; yet these figures made 
rare appearances. Reliability estimates of tests used are somewhat 
more useful than experimenters seem to believe. 
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Comparisons . It might be thought that if questions of samples, 
bases, and measurements had been answered, comparisons would be obvious; 
there were exceptions . Sometimes the lack of careful categorical plan- 
ning in any other department showed Itself here. One study without a 
control group, and for which normal grade level at the end of a semester 
should have been the base, found the investigators comparing pre- and 
posttest scores, and finding them significantly different; a reasonable 
translation of their findings would have been "the hypothesis that our 
methods produce no results can be rejected" — hardly an important con- 
tribution. 

Comparisons were made, among others, of: 

- dissimilar groups ; 

- pre- and post-tests, with severe attrition in the sample between 

the two; 

- experimental and control groups with the same post-tests but 

different pre-tests ("adjusted") . 

Significance . Even when reasonable demands had been met elsewhere, 
the portions on significance testing were a constant source of frustra- 
tion; there would be no test of significance when it was obvious that 
the basis for it existed; there would be medians or means but no 
standard deviations — or if by some chance these had been included, the 
correlations or covariances between pre- and posttests were missing; or 
the size of the sample would be unknown. 

Occasionally all intermediate statistics would be given but there 
would still be the minor headache of deciding for small samples whether 
they were statistics (i.e. , precise figures for the samples without N 
"correction for bias") or estimates of parameters (i.e., with the — =- 
factor) . 

Worst of all, there would be significance figures with no support- 
ing data or statement of the test used, or even whether the test was 
one- or two-tailed. There were even cases where it was known from 
the reports that F-ratio tests had been used yet it was not certain 
that they did not refer to the variances of gains which would not mean 
that the differences between means was significant. 

There were even times when it was not clear what the investigators 
were doing, and there was a suspicion that they did not know either. 

A simple t-test is most often appropriate, though most authorities 
believe it should follow upon an F-test. However, when covariance 
analysis is appropriately done, an F-test is sufficient; but the suit- 
ability of covariance analysis is not at all easy to decide on. Certainly 
it is appropriate when adjustments for reasonably small starting differ- 
ences need to be made in several groups — provided the adjustments are 
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not needed because the tests are different. If they are, the whole 
idea is, to say the least of it, debatable, for then concepts of pre- 
diction and regression are brought in where that of equivalence is more 
appropriate. This will be taken up in a little more detail (though not 
much, because this is not a textbook, and less negatively). Here it is 
necessary to emphasize only that a blunt statement of a significance 
level can be misleading; at the very least the test used should be 
specified, preferably with a reference; better still, the basic elements 
should be given to allow one to adopt variations of the test. Above 
all, interpretation would be considerably eased if investigators always 
translated the results of these tests into ordinary language. 

Importance . Almost everyone, these days, is aware of the risks of 
using samples that are too small (although research frequently was found 
based on samples of no more than a dozen or two) . Significance tests 
allow for the sample size, and (apart from the fact that small "samples" 
that are really "clusters” are likely to be representative only of 
stringently circumscribed populations) are more likely to be "fail safe" 
by accepting the null hypothesis falsely. 

There is another risk which cannot really be dismissed in the 
absence of a cost-effectiveness follow-up. The use. of huge samples 
may result, quite correctly in the rejection of a hypothesis of "null 
gain", when the benefits are negligible. Unfortunately it is far more 
difficult to evaluate the importance of a gain than to establish its 
reality. Strictly speaking, one would need to know the cost per head, 
of implementing the discovery, the losses that would continue otherwise, 
and the costs of alternatives — an impossible task for the most part. 
However, in the absence of these refinements, a wet thumb is quite a 
fair wind indicator; the standardized gain gives quite a good perspec- 
tive for common sense decisions (i.e. , gain divided by standard devia- 
tion) . 

For example, physical height can be used as an easy, meaningful 
illustration. It is doubtful whether anything which changed average 
male height by less than half-an-inch would have much of an impact; 
for a few dollars per head it might be considered, but there are more 
urgent Aeeds, at thousands per head. Half-an-inch is roughly a fifth 
of the standard deviation of male height. Yet on control and experi- 
mental samples of 5000, one-tenth of an inch difference between the 
means would be significant at the five percent two-tailed level. 

As a rule-of-thumb, gains of less than a third of a standard devi- 
ation are nothing remarkable. One needs the standard deviation. Too 
often it was not to be found, or even to be derived. 
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Methods of Presenting and Summarizing Data 



Considerable attention was paid to methods of presenting c.nd 
summarizing data collected from reports and site visits. The Office 
of Education requested that descriptions of the programs selected be 
written so that enough detail was offered for a preliminary decision 
to be made in a school district about the desirability of attempting a 
locally modified replication. Since the data was collected from 
various sources and in widely differing forms, this request implied 
the need to reduce the data to a common format in the descriptions. 

„ Th e description writing guide . Each of the writers of descriptions 
on the staff of the study used a description writing guide deve.oped 
during the period of site visits by one of the staff based permanent y 
in Palo Alto. The guide was evolved from a rough draft containing a 
broad enumeration of those details likely to be of significance to school 
district personnel. Later versions were based on experience gained from 
compiling descriptions of three programs subsequently discarded. It was 
found that most of the descriptions could be divided well into nune mam 

parts: 

Introduction 

Personnel 

Methodology : General 

Methodology: Specific Examples 

Evaluation 

Budget 

Quoted Sources 

Sources not Quoted 

For More Information 

Each of these nine parts is explained below. 

In the Introduction a quick overview of the program was provided. 

' First a brief description of the treatment was given, followed by 
details of the pupils served by the program. The Historical deve op- 
ment of the program was dealt with next, rather briefly, and the 
magnitude of the program outlined. Finally, the cognitive behaviors 
measured were listed together with a very short account of the mam 
results of testing. 

The Introduction was intended only to indicate to the reader the 
salient features of the program, enabling him to judge whether he shoul 
read the more detailed description under the other headings. 

The personnel involved in a program were listed in categories. 
After the name of each category (e.g. , educational aides), a few notes 
were usually given in parentheses concerning the qualifications or 
selection procedures for this category. Were these fairly obvious, as 
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in the case of guidance counselors, secretaries, or school nurses, nothing 
was mentioned. Then for each category the more important activities and 

duties were listed, although again nothing was entered for the more obvious 
cases . 

Under this heading there was also noted the time commitment of various 
categories of personnel, particularly if these people were sharing their 
time between several programs. Others only tangentially associated with 
the program being described, such as janitors, cooks, or the district 
director of research in some cities, were mentioned in a final paragraph 
rather than by category. 

The assumptions and objectives of the program were frequently in— 
eluded under a heading Methodology: General, particularly if they had been 
clearly laid down by the program director. A narrative outlining the 
treatment used in the program formed the core of this part, however, in- 
cluding all the major components in as comprehensive a manner as the 
available information allowed. Not only the instructional methods used 
with the pupils, but also any training programs for teachers were dis- 
cussed here. Every attempt was made to provide here a framework of fact 
about what happened in the classroom, onto which the reader might attach 
the additional specific examples of the next part of the description. 

Where specific examples of principal aspects of a program were avail- 
able, these were usually included under a heading entitled Methodology: 
Specific Examples, although some might have been discussed under Method- 
ology: General. The examples were selected from the available publications 
concerning the program, or in a few cases on the basis of personal ob- 
servation of the program. They were selected to illustrate the content, 
methods, or effects of the program $ consequently, they ranged from infants' 
songs to tutors* accounts. Materials found to be particularly useful in 
the program were listed here or described. 

If there was more than one evaluation report available on a program, 
from two sources for the same year or for several years, the Evaluation 
part of the description attempted to deal with each, showing both juxta- 
positions and trends. Hence this part ranged from a few paragraphs for 
some programs to many pages for others. The evaluation reports were * 
examined critically by the staff of this study, and many programs were 
not included in this publication because either the reports or on-site 
inquiries showed that there were no measured benefits of cognitive 
achievement for one reason or another. Even for the programs described 
there were warnings inserted for the reader; in some of the descriptions 
specific comments and caveats were written, as appropriate. 

The measures of achievement used were described first, for each area 
testea, and the test results were summarized ( not presented in full) in 
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tablas. Wherever possible these were presented simply, although the 
level of confidence (p value) for the differences (benefits) was cited 
if known. 

Next under this heading other evaluation indices were discussed, 
usually briefly. The studies undertaken were outlined, and the results 
summarized, to give a fuller picture of evaluation. Should non-cognitive 
achievement, such as improvement of the self-image of pupils, have been 
the «a1or objective of the program, this was mentioned. This portion 
was not intended to be exhaustive. 

The final product of the evaluation (s) was usually modifications 
and suggestions for changing the program. These were discussed last 
under the Evaluation heading. Where possible the reasons for the changes 
were included. From this portion it should be possible to judge what 
changes might be made (were it adopted elsewhere) to improve the program 
still further. 

The Budget heading was not intended to provide for a detailed ac- 
count of all the expenditures associated with a program. Such figures 
were rarely obtainable. Rather it was a description of what components 
a planner would have to take into consideration in replicating the pro- 
gram, together with whatever rough estimates of per pupil costs or of 
global costs the researchers were able to secure. It might include 
typical staffing patterns required, type of materials essential to the 
pro gram , details of space to be provided in schools, cost— trends over 
several years, and other relevant items of this nature. 

Under Quoted Sources were listed the reports actually quoted in the 
description. 

Under Sources not Quoted other publications or documents known to 
the researchers which give further details of the program described were 
listed. 

One or more names and addresses were given for people closely as- 
sociated with the program described or its evaluation. These were 
correct in August 1968, and listed ‘under the For More Information 

heading . 

The result of the use of the description writing guide was that, 
although the descriptions were written by several authors, a common 
format was followed with only minor individual variations. Thus it 
is possible for a person wishing to know about the populations of a 
number of programs to turn to roughly the same paragraph, under the 
camp general heading, in each description. Since not as much information 
is available about some programs as about others, not every feature of 
the descriptions appears in every description. 
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Specimen descriptions drawn up according to this format were sub- 
mitted to the Office of Education, the National Advisory for 

the Education of Disadvantaged Children, and two persons who had been 
principals until recently, for review and comments. When drafts 
21 program descriptions included in Part II had been prepared they were 
mailed to the project directors for correction. Requests for changes, 
mostly minor, were received from about a dozen directors, an n a mos 
all instances were acceded to. 

Granhine the results. A clear graphical summary was sought which 
would amplify the message carried by tables, both for the technical and 
the non-technical reader. It was found that those programs in which 
grade -equivalents had been used to assess status of pupils, in reading 
or arithmetic, for example, a simple graph could indeed be prepare . 

This graph would state obviously the need for disadvantaged children to 
strive towards national norms as a goal, would show the trend of most 
disadvantaged children away from such norms, and would indicate the 
benefits provided by a particular program over a given period. Dia ^ a ® 1 
shows the normative achievement of children taking the nation as a o e. 
The norm may be the subject of controversy, but it is also widely ac- 
cepted as a pragmatic standard. A child achieving at the norm falls on 
the line: at grade 3.0 he has grade -equivalent scores of 3.0. Diagram 1 
also shows the probable normative achievement of badly diaa ^ n ?; a f e |L . 
American children. Repeatedly in this study data occurred which indicated 
that on average the badly disadvantaged achieve at about two-thirds of 
the level of the children at the national norms. Hence the badly dis- 
advantaged child at grade 3.0 has grade equivalent scores of 2.0. Some 
might argue that the ratio is 7:10 rather than 2:3, but that is of scant 
importance. The fact is that the degree of disadvantage greases, 
probably in a fairly constant ratio as shown by the lower line of the 

graph in Diagram 1. 

Having established these two lines as working bases, it is safe to 
predict that the majority of programs for which results are expressed in 
grade -equivalents will have results falling between the two lines. What 
is of interest is whether the individual program is able to induce the 
line for its pupils to trend towards the national norm. This study 
examined examples of programs such as that shown by line C on Diagram 2; 
pupils were actually not learning at all, according to the published 
results, so were becoming quickly even more disadvantaged. 

Other programs yielded results producing line B; in these programs 
pupils were not becoming any more greatly disadvantaged. Line B P a ^ els 
the national norm. To arrest the process may seem laudable, but only 
programs represented by line A could come close to being considered for 
description. Line A shows pupils gaining fast enough to move towards the 
national norm, gaining faster than two months in three. 
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Some of the graphs appear a little more complex because of the 
number of groups represented on them, but all follow the principles 
enunciated. A few show pupils below the disadvantaged norm, a very 
few show achievement above the national norm. 

At simple devices to depict measured benefits of achievement, with- 
out any test of statistical significance, the graphs serve their purpose. 



Diagram 1 
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Diagram 2 

THREE EXAMPLES OF PROGRAM YIELDS 
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RESULTS AND FINDINGS 



Overview 



Altogether, site visits were made to a total of 98 programs, in 
31 locations representing 16 states. From these 98 programs, 21 were 
identified as having met the criteria for acceptance discussed earlier. 
Detailed descriptions of the programs are contained in Part II of this 
report. The remaining programs may not have been selected for a variety 
of reasons, including lack of available data upon which to assess fully 
the program benefits or failure to demonstrate measured benefits of 
practical significance. A major problem in the larger cities, e.g., 

Los Angeles, was the extreme difficulty, or in some cases, the impossi- 
bility of untangling the complex net of interwoven programs to such an 
extent as to permit the tracing of measured benefits to any single pro- 
gram or manageable number of components. 

Table 1 presents a complete listing of the programs accepted, their 
locations, grade levels encompassed, and the number of pupils involved 
during the latest year that data were available. The 21 programs were 
located in seven states plus the District of Columbia. Five of the 
programs were aimed exclusively at preschool children, while two addi- 
tional programs involved preschool plus other grades. As shown in the 
table, the fourteen programs involving elementary school children gener- 
ally spanned four or more grade levels. Only two programs were aimed 
exclusively at one or two grades. Six of the programs selected for 
description covered students at the secondary level, grades seven to 
twelve. Of these, only one spanned the entire range while the remaining 
five each covered two adjacent secondary grade levels, most often in 
combination with one or more elementary grades. 

From the standpoint of the ethnic composition of the pupils, Negroes 
were involved in sizeable percentages in 17 of the 21 programs, while 
children of Spanish surname were similarly represented in 11 of the 
programs. In only three programs were Appalachian or inner-city whites 
represented in significant number. In terms of sheer numbers of pupils 
covered by the programs, the range is extremely wide. Considering only 
the most recent year for which data were available, the numbers ranged 
between 15 and 16,600, with a median of 500. 

Program Components 

Before attempting to summarize the substance of the programs des- 
cribed, it should be mentioned that, while 21 programs were included, 
actually only 20 programs are prime candidates for large scale replica- 
tion. One project. Teacher Expectation in South San Francisco, was 
included primarily to emphasize the importance of teacher expectations 
for the performance of disadvantaged children and to demonstrate the 
impact that such expectations can exert upon actual pupil achievement. 
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Table 1 

SUMMARY OF EDUCATIONAL PROGRAMS DESCRIBED 



Program Title 


Grade 

Level 


Major 

Ethnic 

Group 


N during 
Latest 
Year 


The Preschool Programs of Fresno, 
California 


P 


S 


750 


The Infant Education Research Project 
in Washington, D.C. 


P 


N 


28 


The Early Childhood Project of the 
Institute for Developmental Studies in 
New York City 


P, K-3 


N 


255 


The Perry Preschool Project of 
Ypsilanti, Michigan 


P 


N 


24 


The Diagnostically Based Curriculum in 
Bloomington, Indiana 


P 


W 


15 


The Academic Preschool in Champaign, 
Illinois 


P 


N 


15 


The Homework Helper Program of 
New York City 


3-6,11- 


-12 N,S 


2000 


The Intensive Reading Instructional 
Teams of Hartford, Connecticut 


3-6 


N 


500 


The After School Study Centers of 
New York City 


2-6 


N,S 


13000 


The Self-Directive Dramatization 
Project of Joliet, Illinois 


1-4 


N,S 


134 


The More Effective Schools Program in 
New York City 


P, K-6 


N,S 


16600 


Project Concern of Hartford, Connecticut 


K-5 


N,S 


260 


The Elementary Reading Centers of 
Milwaukee 


4-8 


N,W 


1000 


The School and Home Program of 
Flint, Michigan 


K-6 


N 


2300 
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Table 1 
(Cont . ) 



SUMMARY OF EDUCATIONAL PROGRAMS DESCRIBED 



Program Title 


Grade 

Level 


Major 

Ethnic 

Group 


N during 
Latest 
Year 


The Programmed Tutorial Reading 
Project of Indianapolis, Indiana 


1 


N,W 


' i 

1200 


Teacher Expectation in South 
San Francisco 


1-6 


S 


37 


The Speech and Language Development 
Program of Milwaukee 


1-2 


N,S 


273 


The Communication Skills Center Project 
in Detroit 


2-12 


N 


2845 


Junior High Summer Institutes in 
New York City 


6, 7-8 


N,S 


15000 


Project R-3 in San Jose, California 


8-9 


S 


74 


The College Bound Program of New York 
City 


9-10 


N,S 


3000 



P = Preschool 
S = Spanish Surname 
N = Negro 
W = White 



34 



Because this program had no educational components to replicate, the 
following summary is restricted to the remaining 20 programs. 

Preschool programs . Looking first at the treatment duration of 
the seven programs involving preschool children, we find a near even 
split, with three programs extending for a single academic year and 
four extending for two years. In terms of intensity of treatment as 
measured by the number of instructional hours per week, the programs 
ranged between 5 and 20 hours per week; however, most were the usual 
half-day program five days per week. Ali of the preschool programs 
emphasized language training and the achievement of a pupil to a j t 
(not necessarily accredited teacher) ratio of one to seven or better. 

The active participation of adults, other than professional teachers, 
was a typical cosiponent of the programs described. These adults were 
ordinarily parents or aides or both. Field trips were another component 
present in most of the preschool programs. From the standpoint of 
benefits expected from the preschool programs, most of the programs 
described measured program impact in mental ability or IQ. A variety 
of measures were employed to assess such gains; the two measures us 
most often were the Peabody Picture Vocabulary Test and the Stanford- 
Binet, each being used in five of the seven programs. 

' While the similarities among programs are rather obvious, it would 
be a mistake to assume that the seven programs involving preschoolers 
were all of one mold. On the contrary, there were notable differences 
amnng the progr ams both from the standpoint of underlying philosophy as 
well as in the specific methods employed. In terms of degree of struc- 
ture alone, the programs ranged from almost total freedom for individual 
teacher innovation to the very highly structured, where specific teacher 
behaviors were established to reinforce appropriate pupil responses. 

Elementary programs . As mentioned earlier , most of the programs 
described dealt, in whole or in part, with elementary school children 
in grades one through six. About half of these could be considered as 
extending primarily for a single academic year, while most of the 
remainder were one semester efforts or shorter. Three programs ranged 
between 5 1/2 weeks and 3 1/2 months. Several of the programs were 
directed toward providing assistance after school hours, two involved 
summer sessions, and the remaining programs were aimed primarily at 
providing special education during normal school hours of the regular 
academic year. Because of this diversity, it is impossible to provide 
any meaningful overall summary of the intensity of treatment in terms 
of instructional hours per week. This diversity extends into the very 
substance of the elementary programs, making any attempt to summarize 
program components much more hazardous than for the preschool programs. 

All but one of the elementary school programs measured gains in 
reading achievement. This, of course, was expected since measured bene- 
fits in reading was one of the prime criteria for program acceptance. 

On the other hand, gains in number skills, also a criterion for program 
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acceptance, were assessed in only three of the programs. Despite the 
predominate emphasis upon reading behaviors, the specific instruments 
used for assessing such gains varied widely with only two tests being 
employed in as many as three separate programs. 

In line with this emphasis upon reading, the elementary programs 
of ten -included special reading instruction, using personnel with pro- 
fessional training in this field. Similar to the preschool programs, 
the improvement of the pupil-adult ratio was a major component; the 
ratio, however, appeared to range wider, from one-to-one tutorials to 
one teacher per fifteen students. Also in contrast, the adults involved 
'in the elementary programs were typically professionals rather than 
aides or parents. Emphasis upon parent involvement was much less appar- 
ent than at the preschool level. The component of field trips, so 
popular at the preschool level, did not appear as a major component in 
any of the elementary programs described. 

Secondary programs . Of the six programs that included secondary 
school students, two were strictly summer programs, while one functioned 
both in summer and during the academic year. Of the four programs 
that did operate during the school year, two provided special instruc- 
tion approximately three hours per week, while one provided 15 hours of 
instruction per week. One program was aimed at providing after school 
assistance using high school students as tutors. All of the programs 
attempted to assess program benefits in terms of reading gains, while 
the three programs that were aimed exclusively at junior and senior 
high school students (grades six through ten) also measured achievement 
in mathematics. It is interesting to note that students in grades ten 
to twelve are included in only two of the programs described. 

While half of the secondary programs included might be described 
as remedial reading programs, the remainder emphasize a variety of 
innovative approaches. In this respect, they appear to resemble pre- 
school programs more than elementary programs, particularly in their 
employment of aides and the use of field trips. 
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CONCLUSIONS AND RECOMMENDATIONS 



Guidelines for Program Design and Evaluation 

Looking back from the vantage point of accumulated experience in 
retroperspective, if the word is permissible — there are aspects of 
the programs studied that stand out and call for comment, much of it 
favorable, and some that prompts positive suggestions with constructive 
^intentions. As always, research designs can take on a variety of 
shapes even when the same objective is in view. Nevertheless, a firm 
framework with common features can almost always be found; the super- 
structure is more varied,* yet even there, with no clear line of de- 
marcation, it would seem to be molded to suit some point or range 
on a continuum from "Proof" research to "Enquiry." xhat is to say, 
some patterns suit the purpose of searching for and formulating 
hypotheses in areas where little is known. Rather different forms are 
adopted where a substantial body of information already exists, with 
a network of known and inter-related facts and with selected and 
fairly clearly defined areas of search. The choice of statistical 
techniques reflects the objective. 

Psychological research, for example often leans towards the "En- 
quiry" end, and makes a good deal of use of such devices as partial 
and multiple correlation, or cluster or factor analysis. These were 
seldom encountered in this investigation. By far the greater proportion 
of the projects examined were manifestly intended to get plain ' Yes 
or No" answers; there were comparisons between groups, or with population 
norms using measures of central tendency, with t-tests and occasionally 
chi-square; even when F-tests were applied it was more often to the 
results of covariance analysis, embarked on as a refinement which allowed 
for differences in starting values between experimental and control groups. 
This will be discussed further later. 

For the skeleton of this common theme, names have already been given 
to some of the more important bones in an attempt to codify difficulties 
encountered. There is perhaps a slight risk that some may interpret this 
as a preference for a rigid format along the lines of propositions in 
ancient Greek geometry; in fact there is more to be gained than lost in 
such formality as Euclid’s. Nevertheless, there is room for individuality 
as long as essential information is supplied, and supplied explicitly 
rather than leaving it to readers to unearth, deduce, or interpret. 

Basic to some reports were several acceptable alternatives, and it is 
encouraging to have support for this stand from some influential bodies. 
An example which made for very easy reading comes from the Center for 
Cognitive Learning of the University of Wisconsin. Their technical 
reports make use of these headings, although they modify them as needed: 
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Background — With soma account of personnel involved, ac~ 
co mmo dat ion , the population aimed at, related studies 
and references, special difficulties encountered and the 
general purpose of the project. 

Subjects — This covered the sample drawn, its chief features 
and the method of drawing it . 

Treatments — A detailed description was given of the method 
being tested and of the materials used. 

Procedures — This covered organizational details as well as 
the provision for comparisons. 

Data Collected — This included short descriptions of the 

variates as well as their place in relation to specific 
hypotheses to be tested. 

Analysis of Data — The statistical techniques adopted were 
given and the data to which they were applied. 

Discussion — Here, a few important tables and charts, but 
most important, a step too infrequently taken by 
investigators, a translation into English of the 
statistical conclusions. 

The United States Department of Health, Education, and Welfare 
used these headings and sub-headings for their table of contents in at 
least one of their reports! 
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Procedures 

Design 

Criteria for selecting subjects 
Assessment and diagnostic techniques 
Screening 
Repeated measures 



For one last and slightly different example, we drew from one of 
the many Center for Urban Education evaluation reports. They used 
quite a small number of headings and sub-headings, but very efficiently: 

Introduction 

Description of program 
Objectives 

Evaluation Procedures 

Objectives of evaluation 

Selection of schools and classes for evaluation 

Data collection 

Bases for evaluation 

Instruments 

Analysis of data 

Orientation of examiners and observers 
Findings 

Conclusions and Recommendations 



Within such frameworks as these (and there were several others 
equally good) it was fairly easy to find one’s way directly to each of 
the focal points. However, this facilitation for readers or users is 
not the only argument in favor of making the design fomal; failure 
to check againsc a list of essentials is a source of oversights and 
blind-spots; worse, it allows loose planning with shifting definitions, 
side-tracking, and inappropriate offshoots. Good, if personal, check- 
lists are as useful for research design and reporting as they are for 
traveller's luggage, and some years ago Symonds proposed ”A Research 
Checklist in Educational Psychology" (1956). He intended it for students 
who were preparing dissertations, but why should qualified investigators 
be less systematic? Symonds expands on these, his main headings: 
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A. Scope and Definition of Study 
5. Hypotheses 

C. Background 

D. Definitions 

E. Method of Study 

F. Design 

G. Sampling 



H. Studying Personality 

I. Tests and Measures 

J. Use of Judgment 

K. Content Analysis 

L. Statistical Handling of Results 

M. The (Ed.D.) Report 



For those who have not yet discovered it, the "Handbook of Research 
on Teaching," edited by N. L. Gage (1963) is a miniature encyclopedia 
(1,200 pages by about 30 authors). It is difficult to pick out the 
most valuable chapters; most rf them contain profit, but particularly 
relevant here is that by A. A. L^^isdaine, Chapter 12, "Instruments and 
Media of Instruction," pages 5o3 to 682; that author gives his purposes 
as " .... to aid the research worker in sharper, more useful delineation 
of problems; (2) to analyze the current status of research-based knowledge 
in the more important problem areas; and (3) to consider methodological 
techniques an pitfalls in order to improve the incisiveness and efficiency 
of future research." 

There were three more specific areas for which more careful pre- 
paration by investigators is recommended: 

1. Problems of sampling 

2. Study of content of tests used 

3. Statistical analyses using 

(i) Medians 

(ii) Analysis of covariance 



Problems of sampling . In spite of categorical warnings by statisti- 
cians over the years, most of us at some time or other ignore limitations 
upon the use of tests of significance set by sampling methods, perhaps 
because most of the authorities put their caveats in much too ponderous 
mathematical terms. Without the need to Justify the objections or to 
provide corrections, they can be put quite simply: 

Significance tests, suitable for those of us whose expertise lies 
in other professions than Statistics, are derived from theoretical models 
which depend upon the simplifying assumptions implicit in random sampling ; 
and in turn this latter concept is more easily understood than defined. 
"Randomness" today carries a long list of necessary conditions, rhe most 
important of which is that all individual cases in the population must 
have an equal chance of being chosen . This condi » Lon is violated when 
cluster sampling (e.g., selection of complete classes, schools, or areas) 
is adopted and this is the obvious method in education. Simple t-tests 
do not have simple interpretations when this is done. 
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Ai. by-products, those who feel that they could benefit from another 
look at the old textbooks can expect to tighten their grasps on concepts 
of population and statistical inference by reading "Sampling in a Nut- 
shell" by Slomim (I960) . This should be followed by the progressively 
harder publications from Stuart (1962) , Yamane (1967) , Deming (1950) , 
and Cochran (1963) . ; 

r 

Study of content of tests used. It is easy for most investigators 
to describe the content and essential features of their own aims, materials, 
and methods; some translate the results of the statistical analysis into 
ipore meaningful English, but there is a good deal of room for improvement 
in the discursive treatment of the content of the measuring instruments 
used. Failure to do this leads to some deceptive and unwarranted conclusions. 

For example, some tests call for elements of interpretations, or 
subjective racing or assessment; or they are susceptible to subtle in- 
fluences of sympathy, familiarity, or methods of communication. Certainly 
statistical treatment will reflect this, but explanations of changes in 
numbers sometimes reflect wishful thinking, where consideration of test 
content could have given simpler, if less dramatic explanations. Almost 
invariably, the sampling process precedes the testing; recent experiment- 
ation has shown that this can easily produce startling differences in 
results via some quite ordinary biasing mechanisms. We not only see what 
we want to see - we even encourage responses which give superficial but 
spurious confirmation, and can do this quite unconsciously. Rosenthal 
and Jacobson (1968) have written a book which ought to be read before the 
start of any new project. Some of the experiments they describe show what 
can be done even with animals much lower on the scale than Homo sapiens; 
and both real and spurious effects can be produced by "expectations," 
thus recalling the so-called "Hawthorne effect." The work of these two 
. authors will have an impact on experimental design, on treatments, on data 
collection, and on interpretation of results. 

There is another reason for careful scouting and discussion of test 
content, and that is that where names are insufficient descriptions of 
content, changes may derive from undescribed and even otherwise unimportant 
parts, leading to misinterpretation of the results, at the very least. The 
most frequent example of this is the uncritical use of tests of Intelligence 
to demonstrate gains as a result of some new component, occasionally with 
a thinly- veiled suggestion that perhaps there are no real genetic dif- 
ferences and that, after all, all men really are created equal. 

The criticism is often made that intelligence tests do not measure 
intelligence, based on the fact that they do not measure it directly; 
there is nothing unusual about that — not only the behavioral sciences, 
but even the so-called exact sciences are often driven to inferred 
measurement. Physicists have long used the galvanometer to measure 
electric current by forcing it to generate a magnetic field, and this 
in turn to produce torque. ' Or it is said that the tests are suitable for 





a narrow band of the culture which produced them; again there are parallels 
from physics, where a mercury thermometer breaks down completely at both 
very low and somewhat high temperatures. Extraneous influences affect 
the results of both intelligence measurement and barometric measurement 
of altitude (humidity in this case). Last, and most important, psycholo- 
gical instruments, like those of physics, can be pressed into unusual and 
less appropriate service by capitalizing on some minor aspect or feature 
of the instruments. 

Several intelligence tests, including the Stanford-Binet and the 
^Wechsler batteries, have sub-tests of vocabulary based, rightly or wrongly, 
on the assumption that spontaneous acquisition of a vocabulary is a 
reflection of the individual's basic capacity for learning. Anything 
which reduces the spontaneity (e.g., gifts of storybooks, or remedial 
teaching of reading) will produce changes localized in the performance of 
this sub-test, whose scores are part of the total. The explanation of 
"gains in IQ" are then essays in the obvious; the inclusion of sub-tests 
of arithmetic, or form relations, or of picture completions, arc as 
likely as not, irrelevant clutter, a waste of time; the result of untidy 
thinking and the cause of false conclusions. It is essential that 
investigators consider the instruments they use, in details of under- 
lying assumptions, mechanisms, discrete parts, and scoring weights. 

Statistical analyses: use of medians . The median score, or at least 
its close approximation, is a little easier to find by hand than other 
measures of central tendency; you can get it as long as ycu can count, 
even if you can do no arithmetic. That is just about its only advantage, 
and after all who does things by hand any more? It does less than 
justice to interval measurement and is appreciably more unreliable 
than the arithmetic mean to say the least (its standard error is 25 per- 
cent larger). Worse still, strong arguments have been produced in this 
A. I.R. team to support a hypothesis that bimodal trends in the distribu- 
tions of both independent and dependent variates will be found in many 
of the projects being considered; and bimodal distributions can make the 
median a deceptive statistic, as shown in Diagrams 3 and 4 . 



Diagram 3 

AN EXAMPLE OF NO CHANGE IN MEDIAN BUT A CHANGE IN DISTRIBUTION 
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Diagram 4 

AN EXAMPLE OF A CHANGE IN MEDIAN AND DISTRIBUTION 



Prpfpflf RMttpr! I III 


/ 1 IUJ 


i 


low 


V 

Median 


high 


Pnc *- fro a #- RPaffpr! 1 Iff 


i "j 


1 Mil 1 


low 




high 



However , if gains are going to be quoted in terms of medians , there 
is no excuse for the failure to test the significance of such gains. 

All that is necessary is first to pretend that means were used and to 
apply the appropriate calculation of t; then divide this t by 1.253 
(i.e., by ?2) , and refer this adjusted value to t-tables. The only 
incongruous element is that to calculate t, one needs the variances) 
which in turn need the arithmetic means. Better still) avoid the use of 
the median. 

In passing, a related criticism is concerned with the use of centiles 
(often incorrectly referred to as "percentiles") which were sometimes 
found to be "averaged." 

Statistical analyses: covariance analysis . No single statistical 
device appeared more frequently than this , for several reasons , In the 
first place, the commonest design involved the use of pre- and posttests 
of the same groups and the means ware therefore correlated within groups; 
so covariances were to be used in any computations • Secondly , for those 
studies which used control samples in addition to experimental, there 
were often starting differences which had to be allowed for before com- 
paring gains, and again correlations (and therefore covariances) 
have to be considered. Thirdly, and it is here most of all that issue 
can be taken, some projects, of necessity or from choice, used different 
tests, not only for starting and finishing, but as pretests for the two 
comparison groups; now gains have to be calculated after converting one 
or other of the reference tests, and starting differences can be de- 
tected (and allowed for) only after equivalences have been established. 
Even two intelligence tests like the Stanford-Binet and the Wechsler 
differ in their standard deviations, so that group means appearing on 
either side of the population mean of 100 will have distances expressed 
in different numbers of IQ points; an IQ of 130 on one system could 
easily correspond to one of 136 on the other. Here "prediction via 
the regression line would appear (falsely) to be involved; the correlation 
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coefficient is part of the regression coefficient, and covariance is 
part of correlation. The attempt to use "predicted" values of one test 
from obtained scores on another is wrong in this context ; equivalence 
not prediction is what is needed. The argument can become quite 
involved and technical, and there is room here for an interpretive 
approach only. 

The rationale upon which regressional prediction rests, is that, 
corresponding to every point on the continuum of the independent variate, 
there is a range of empirical values of the dependent variate, and one 
can avoid the largest errors by forcing one's prediction to be the mean 
of this empirical array. This array mean is a point on the regression ^ 
line, and the function of the correlation coefficient is to "conservatize 
the prediction. To use an analogy from baseball, one can reduce the 
risk of being "tagged" by always staying close to a "base," even at the 
cost of missed opportunities. That is why there are two regression 
lines, with a choice dependent upon whether one is departing continuously 
from A towards B , or cautiously from B towards A. One of the direct 
results of this conservatism is the shrinkage of the scatter of "pre- 
dicted" values by comparison with that which would have obtained from 
direct measurement, and the shrinkage is in inverse rexation to the size 
of the correlation coefficient. Thus for a representative sample with 
a real standard deviation of 15 IQ points on the WISC test, the standard 
deviation of scores "predicted" from the Stanford-Binet could be as 
low as 10 IQ points if the correlation between the two was .7. Notice 
then that the variance ratio (F test) for pretests (one actual and one 
"predicted") for two groups that were in fact identical would be 2.25. 

The position can be even worse if the regression lines are derived from, . 
say, population values, and applied to samples which are not representative, 

even the "predicted" mean can be wrong: 



Diagram 5 

THE RELATIONSHIP BETWEEN EQUIVALENCE AND REGRESSION LINES 
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For a sample whose Stanford— Binet IQ's ranged from a to c, with 
a mean of b, their actual WISC values would have ranged from i to g, with 
a mean at h . We would have predicted a sample mean of e, on a range 
from f to d. 

The discrepancy between the two systems stems from a difference 
in underlying assumptions. For predictive systems* we assume that two 
tests measure a common source of variance, as well as two real but not 
common sources, one unique to each test. In the equivalence problem, 
we must assume that differences between scores are the result of random 
error in either or both and of scaling only. 

There are several alternative approaches any one of which will give 
less misleading results than regression; the simplest is to use the major 
axis of the bivariate scatter obtained empirically for a sub-sample 
representative of the two samples. Congruent solutions have been pro- 
duced repeatedly over the past 30 years by such eminent men as Sir Cyril 
Burt in Britain and Frederic Lord in U.S.A. , among others. 

The real point, however, is this: analysis of covariance is a "real 

high-falutin'" process. The justification for its use, more often than 
not, is an attempt to compensate for errors and deficiencies in sampling, 
or in measurement, which were best avoided in the first place. It can 
add refinement to the analytic process by making fine adjustments for 
small discrepancies which are quantitative only. Large discrepancies 
may well conceal qualitative differences in which case the procedure 
may be inappropriate to say the least. Analysis of covariance is not 
such universal application as its use seems to imply. Sophistication of 
statistical analysis is never a compensation for deficiencies in design 
or measurement, and still less (if possible) for careless sampling. Where 
errors have been allowed to creep in, the only correct procedures or 
adjustments are those that make the conclusions to be drawn more con- 
servative; and the grosser the nature of the data, the simpler should 
be the significance tests applied. There are some quite simple t- and 
chi-square tests which could take the place of some covariance analysis; 
leave the latter to those who are prepared to write a chapter justifying 

its use. 

In summary, there are clear guidelines in existence for the design 
and evaluation of programs. Some of the most significant have been 
discussed here. 

Two short bibliographies may be added, dealing with sampling and 
equating, both vital aspects of design and evaluation. 
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Approaches to Cost-effectiveness Analysis in Compensatory Education 

The debate about the feasibility of applying cost-effectiveness 
analysis to education has gone on for only a few years, but it has 
been a loud one. As noted earlier (in the Introduction to this report) 
the proponents of cost-effectiveness analysis claim that expenditures, 
even in education, must be justified by hard evidence rather than 
opinion, and that such hard evidence must be pursued relentlessly to 
provide the basis for cost-effectiveness analysis, which will ensure 
the most efficient allocation of available resources. Their opponents 
declare that hard evidence will never be available anyway, so the 
notion of applying cost-effectiveness analysis to education should be 
abandoned now. The proponents have yet to produce proof because few 
suitable models have been evolved as yet and none put into action. 

Thezr opponents can only point to the "soft" evidence now available, 
without proving that hard evidence is absolutely inaccessible. The 
debate is likely to be continued for some time. This report provides 
ammunition for both sides. 

Cost effectiveness analysis, properly speaking, seeks to establish 
the cost-effectiveness ratios of each of a series of possible alter- 
natives, to assist in the choosing of one of them. Given a single 
clearly defined objective (say, over three months, the raising of 
reading grade— equivalents on the Metropolitan Achievement Tests by an 
average of ten months among a sample of 100 Harlem third— grade children 
reading one to two years below the national norm) , we might suppose 
that there would be certain alternatives of instruction that could be 
employed. Among these would be individual tutoring by peers reading 
at or above the national norm, group tutoring by these same peers, 
individual or group tutoring by college students, or by teachers, dur- 
ing school or after school, teaching by computer, learning from pro- 
grammed courses, and so on. The number of possible alternatives, each 
clearly defined, is obviously great. Theoretically, each could be 
costed, and a final set of ratios produced, each ratio showing the 
benefit (as stated in the objective) in relation to the cost of a 
particular treatment. Then a choice among the treatments would be 
feasible; normally the one costing the least would be chosen, although 
a complete cost-effectiveness analysis would also attempt to take the 
wide view, in the sense of allowing for side— effects of many kinds* 

It is clear that for the kind of analysis just described consider- 
able data are required concerning both inputs (costs and treatments) 
and outputs (achievements) . None of these may be simplistic or over- 
generalized; each must be identified and defined separately. Each bit 
of information must be demarcated from the next bit, lest the categori- 
zation of bits lead to faulty generalizations. In practice, in educa- 
tion at least, such identification, definition, and demarcation is not 
carried out. Costs, treatments, and achievements, none of these is 
analyzed in sufficient detail at present. This is not to imply that 
they cannot be analyzed, but it does imply that cost-effectiveness 
analysis is not feasible in education until these are analyzed. 
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If the objectives of educational programs are examined (and they 
in turn lead to achievements, the output), straight away the immaturity 
of the analytical system is revealed. The apparently clearly defined 
statement of an objective given above may well be inadequate for cost- 
effectiveness analysis, for all the fact that it is more clearly 
stated than most. Reading grade-equivalents are wide open to discussion 
as meaningful measures; different levels of the Metropolitan Achieve- 
ment Tests (several of which might be used in this sample of 100 Harlem 
third-grade children) differ quite radically in the items they contain; 
an "average" of ten months could conceal a good deal about the extremes 
of the distribution; no basis was suggested for selecting the sample; 
third-grade in Harlem is not a valid indicator of anything but physical 
contiguity, and so on. In other words, one could not be certain about 
the exact nature of the output, and it would be quite wrong to attempt 
to generalize cost-effectiveness analysis based on this output to other 
apparently similar situations. Instead of an absolute analysis model, 
a probabilistic model has to be used, by which estimates are made of 
the output given variations of this kind or that. This becomes a much 
more complex process, with results less comprehensible to or useful for 
administrators . 

As soon as more than short-term objectives are introduced, the 
definition of output becomes even more complex. Raising reading grade- 
equivalents during the third grade is relatively simple as an objec- 
tive, but not if the long-term effects of doing so are to be taken into 
account. The output, over the long term* then becomes somewhat different 
from the objective, which was short term. The effect of raising reading 
age in the third may not have died out by the twelfth grade, or even 
beyond. Such consequences are incredibly difficult to identify, let 
alone quantify. (Drop-out rates, crime rates, college admission rates, 
marriage rates, and birth rates may all be more or less affected.) Yet 
they should be taken into account, since two alternative treatments may 
not produce the same lasting effect or long-term benefits, in which case 
the choice of one or the other will be influenced. 

The definition of treatments poses another series of difficulties. 
Both intensity and quality of instruction have to be taken into account. 
Intensity is more easily arrived at , through attendance rates, hours of 
schooling, pupil/teacher ratio, and the like. Quality of instruction 
can be described from observation or measurement. The former provides 
the better account of the items used in instruction and their relation- 
ships, but offers merely a subjective assessment of quality. The latter 
takes note of the products of instruction but ignores the process very 
largely. Even these rough tools, however, show wide variations in the 
quality (and nature) of instruction from one classroom to another, no 
matter whether the teacher is held constant or not. To summarize, one 
cannot be sure of the treatment aspect of the input. Again, sweeping 
generalizations are hard to avoid, though they should be avoided. 

The cost aspect of input is perhaps the one on which there is most 
immediate hope. The relevant prices of goods and services are what are 



sought. To be relevant, these prices must allow for anticipated changes 
in relative prices of items, although not for expected changes in the 
general price level (since these would be computed in any case for all 
alternatives); they must allow for all overhead or indirect costs; and 
they must omit non-essentials. All three of these requirements are 
hard to meet if one is examining costs of education as it is at present 
organized; While it may be possible to predict increases in teachers 
salaries relative to increases in the costs of textbooks, it is not ^ 

so easy to anticipate accurately social pressures which place teachers 
aides in classrooms (or remove them) or which provide, in the more 
distant future, transistorized television sets for satellite trans- 
missions in every rural community. Overhead or indirect costs may be 
available as some gross figure which has to be broken down to discover 
which of its components refer to the treatment under consideration. 



The long catalog of problems just described is not peculiar to cost 
effectiveness analysis in education, however. Prest and Turvey (1965), 
in a study supported by the Rockefeller Foundation, provide numerous 
examples from other fields in which similar problems are encountered. 
They cite as a typical case the work done during the planning of the Ml 
motorway (freeway) in Britain by Coburn, Beesley, and Reynolds (1960), 
Reynolds (1956)* and Foster (1963) : 

The calculation of net annual savings was classified under 
four heads: (i) those relating to diverted traffic; (ii) those 
to generated traffic; (iii) savings in non-business time; and 
(iv) the effects of the growth of Gross National Product. Under 
(i) (diverted traffic) estimates were made of the likely net 
savings of the traffic diverted from other routes to the Ml, i.e., 
positive items, such as working time savings of drivers, 
vehicle-usage economies, petrol savings, accident reductions, 
etc. , together with negative items in respect of additional 
mileages travelled on faster roads and maintenance costs of the 
motorway. 

In respect of generated traffic the argument is that the 
opening of the motorway would in effect reduce the "price" iin 
terms of congestion and inconvenience of motoring) and enable 
demand which had hitherto been frustrated to express itself 
in motorway usage. As it must be assumed that benefits per 
vehicle-mile to frustrated consumers are of less consequence 
than those to actual consumers (if not, they would not re- 
main frustrated) , they were rated as half as great as the 
latter in the Ml calculations. 



Savings in non-business time were the third main in- 
gredient. This calculation involves many complications, to 
which we shall return in a moment. The fourth component was 
the introduction of a trend factor, to allow for the long- 
term growth of Gross National Product and the effects on 
the demand for road travel — an obvious ingredient of any 
calculation, whether relating to private or public invest- 
ment. The upshot of the combined calculations was that the 
rate of return was of the order of 10-15%. 

A number of comments can be made on these calculations. 
First, there are the obvious statistical shortcomings which 
are recognized by everyone, including the authors. Second, 
there are a number of minor omissions, such as allowances 
for police and administrative costs, the benefits accruing 
to pedestrians and cyclists, etc., the advantages of more 
reliable goods deliveries. Thirdly, there are some incon- 
sistencies in these particular calculations, in that on, some 
occasions a long— period view seems to be taken (e.g. , when 
calculating the savings resulting from reductions in road 
vehicle fleets) and on others a short-period one (e.g., 
in assessing the benefits of diversion of traffic from the 
railways). Much more important than these points are the 
savings due to accident reduction and to economies in travel 
time, where important logical and practical issues arise. On 
the first of these, the economic benefits of a fall in the 
amount of damage to vehicles and to real property, the work 
done by insurance companies, the work of the police and the 
courts are simple enough. It is the loss of production due 
to death or, temporarily, to accident or illness which raises 
complications. However, these complications are exactly the 
same as those raised in cost-benefit studies of health pro- 
grams, and so it will be convenient to leave discussion of 
this general topic until we reach that heading. 

This leaves us with the problem of valuing time savings; 
as these savings often form a very high proportion of total 
estimated benefits of road improvements, they are extremely 
important. Unfortunately, these calculations have not so 
far been very satisfactory. 

Whatever the valuation procedure followed, it is necessary 
to assume that one time saving of sixty minutes is worth the 
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same as sixty savings each lasting one minute, since 
estimates of the value of time savings of different lengths 
are unobtainable. On the one hand, it is clear that some 
short-time savings are valueless, since nothing can be done 
in the time saved. On the other hand, however, there are 
cases where tl e extra time makes possible some activity 
which would otherwise be precluded, as, for instance, when 
arriving a little earlier at a theatre means that one does 
not have to wait until the interval to gain one’s seat. 

Similarly, the value of an hour gained may depend partly 
upon when it is gained. Faute de mieux , such variations 
have to be ignored and an average treated as meaningful. 

Prest and Turvey conclude by saying, with regard to this case: 

So at the very least, one can say chat there are major 
unknowns which may or may not prove tractable to further 
analysis . 

Many studies of cost-effectiveness in education have taken a much 
brohder view, in an attempt perhaps to avoid the limitations imposed 
by more rigorous definition. For instance, there have been those which 
have concentrated on the costs of higher education, including the loss 
in income suffered by students while studying, and the benefits to be 
accrued in the form of higher incomes after graduation. Becker (1964), 
Hansen (1963), and Hunt (1963) deal with such comparisons. Hunt applied 
multiple-regression techniques in an attempt to detect relationships 
more accurately, but Prest and Turvey (1965) remark that the confidence 
intervals of his estimates of the effects of various variables were very 
wide, and it was made abundantly clear that correction for other deter- 
minants of income was very necessary. Blaug (1965) considered such 
difficulties in some detail. 

Other studies, such as that by Denison (1964) , have examined the 
effect of education upon the national product. Denison calculated that 
education in combination with a factor called ’’advance of knowledge 
accounted for about two-fifths of the growth of real income in the 
United States from 1929 to 1957. In these studies, education is viewed 
as a process of investment in people. The payoffs are discussed by 
Weisbrod (1962) who sees them in broad terms of increasing production 
possibilities, reducing costs (thereby making resources available), and 
contributing to the general welfare. Spiegelman et al. (1968) have 
added another: increase in benefits which increase an individual s 
enjoyment of life. 

The Spiegelman et al. study (1968) examined not only the increase 
in personal income due to graduating from high school and going to 
college, but also intergeneration benefits (those accruing to the off- 
spring of the generation currently being educated) , and the reduction 
of juvenile crime. These three selected outputs were related to inputs 
paid for by ESEA Title I funds; they were not intended to be comprehen- 
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sive and others could certainly hcve been selected. The study devel- 
oped a benefit /cost model along fairly generalized lines, analyzed the 
benefits to be expected (in terms of direct increase in attainment, 
probability of high school graduation, probability of going to college, 
intergeneration effects, and probability of being arrested for committing 
a juvenile crime), and computed the returns to education. Equations 
were derived which attempted to take into account a number of consider- 
ations of the kind discussed earlier in this section. 

The Technomics (1967) report contained much of value in the setting 
up of cost-effectiveur js analysis in education. The Technomics team 
discussed very succinctly the data required to accomplish such analysis, 
and made every effort to secure the data itself, so that their model could 
be used. The team investigated the ESEA Title I projects of 12 cities 
to discover what costs, treatments, and achievements there were. 

The conclusions reported were tuat "it is feasible to apply a suitably 
modified version of cost /effectiveness technology to matters of educa- 
tional expenditure; furthermore, we can describe the necessary modifi- 
cations in considerable detail. We can also specify an easily under- 
stood tool for program planning and budgeting that will be immediately 
usable at the local level." This optimistic viewpoint was indeed 
supported by a model, which employed two special techniques. The first, 
cluster analysis, was claimed to allow the user to associate benefits 
(the achievements) and costs with the characteristics of a learning 
environment. The second, a matrix, was employed to define and clarify 
the structure of the "educational possibility space." 

The Technomics team was certainly not guilty of a facile approach, 
but its optimism may be questioned. The development of the model in- 
volved the setting up of a large number of categories, into which simi- 
lar but sometimes differing items were compressed. To do so required 
the of various untested assumptions. When it came to applying 

data to the model, further compressions were necessary on account of 
the nature of the data. In spite of the fact that Technomics was able 
to survey 22 projects in 12 cities, which accounted for expenditures 
of nearly half a billion dollars, the number of cases proved too limited 
to provide adequate information on many variables. Low information 
variables were therefore eliminated from the treatment, at unestimated 
costa to the model as a whole. This much was admitted in the report. 

A lis t of project variables was derived, the scope and limitations of 
which are best depicted by the list itself: 

Al. Focus on child (identifying and meeting special needs) 

A2. Kedefinition of learning 
A3. I n ser v i ce training 
AA. Administrative staff support 
AS. Teaching staff support 

A6. Cooperation of teaching and administrative staff 
A7. Services 
AS. Methodology 

A9. Attention to the child in class 
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A10. Freedom to experiment with teaching methods 

All. Parental involvement 

A12. Community involvement 

A13. Project planning 

A14. Class size 

A15. Classroom management techniques 

A16. Breadth of program (number of levels of instruction) 

A17. Methods of selection of pupils for the project 
A18. Racial mix of project group 
A19. Racial mix of school 

Few of these variables stand up well to close examination. Most 
require subjective definition subjectively applied, thus adding to the 
qualitative element of the base for an apparently quantitative analysis, 
and thereby increasing the chances for error. The benefit variables 
suffer from the same problems. 

Cluster analysis, one of the technioues mentioned earlier, again 
depends on compressions. To group similar variables, judgments have 
to be made about their similarity. As the Technomics report indicated, 
cluster analysis is used to reduce the number of variables that must be 
handled in the statistical analysis while minimizing the loss of infor- 
mation entailed by the reduction . Probabilities once again enter the 
model, rendering the final conclusions less trustworthy. 

The views of the Technomics team are the more remarkable in the 
light of the comments in the report concerning the difficulties expel 
ienced in data collection, difficulties equally experienced by the 
AIR team two years later (as described under Methods and Procedures). 

To quote one relevant portion from the Technomics report: 

Most damaging of all, however, was the use of inappro- 
priate evaluation procedures. Related measures were commonly 
tested as though they were independent; the possibility of 
multiple causation was ignored in testing gains; control 
groups were almost never used to provide either baseline 
or comparative data. Projects were allocated to schools 
on jases absolutely unrelated to evaluation design needs. 

The chances of producing a useful cost-effectiveness analysis based 
on data from such basic errors, even with a fine model, seen remote. 

The predictions made by Technomics to the effect that the data would 
soon improve as planning (particularly under Title I) matured, have 
not been borne out by AIR experience. Projects still do not have clear 
and workable objectives, benefits to be derived appropriately stated, 
and achievements accurately measured. 

Technomics made many good suggestions, however, for improving sys- 
tems and evaluations so that better dara would be available in the 
future, to apply to the Technomics model. (Similar suggestions appear 



under Guidelines for Program Design and Evaluation in this report.) 

Most of the changes related to evaluation and accounting procedures, 
rather than to the description of the nature and quality of treatments. 

In concluding this discussion of the Technomics study, it must 
be emphasized that the Technomics report repeatedly discussed the diffi- 
culties. What is puzzling is the belief of the team that a viable tool 
been developed Perhaps evidence of its viability will be forth- 
coming soon from some project which follows the Technomics recommenda- 
tions and uses the program planning and budgeting matrix, the planning 
vectogram, and cluster analysis. 

The most critical difficulty in the way of cost-effectiveness 
analysis in compensatory education at present is one scarcely mentione 
in the Technomics study, but one which became very clear during this 
survey. In order to be able to select from alternative treatments, it 
is essential that each of the treatments carry some guarantee that it 
will result in the anticipated output (benefit). Seven different treat- 
ments at differing costs, may be presented as means of reaching a parti- 
cular goal, say raising by one year the reading grade-equivalents of a 
certain group of pupils. All seven must have beeiT shown to have been 
successful with similar pupils elsewhere. Evidence that alternativ e 
su ccessful treatments exist is not available at present among compen- 
satory education programs in the United States . Even if the programs 
with measured benefits of cognitive achievement which were not studied 
during this survey were included, it is very doubtful whether there 
would be enough examples available to begin cost-effectiveness analysis. 
Comparisons between treatments which have yielded differing amounts of 
the same type of benefit might just be possible, but would be fraught 
with hazards on account of the non-addit iv e quality of some benefits. 



Thus the first step towards cost-effectiveness analysis must be 
through improving compensatory programs until the treatments can be 
said to be successful. At present the cost-effectiveness choices lie 
almost entirely among the more or less unsuccessful, which makes nonsense 
of cost-effectiveness analysis at all. 

The view taken in this A.I.R. study is that cost-effectiveness 
analysis in education, in the sense of providing cost-benefit rat *f® to 
assist in choices among alternatives, cannot yet be undertaken. The 
Technomics model cannot be applied successfully, nor can the much more 
theoretical one of Abt (1967), until improvements are made in programs 
of compensatory education, and in the schools' systems of recording and 
reporting inputs and outputs. The improvements in the programs s*J 
assisted by this study, in that the existing successes are advertised 

more widely. 

% aove towards improving the schools' systems does appear to be 
feasible, and the current (1968) GE-Tesipo study nay be assisting it. 

The A.I.R. survey has shown repeatedly that the najorlty of programs 



have been set up as demonstrations, not experiments, including a 
variety of components which local educators confidently expected would 
improve the education of the children involved. A few programs were 
specifically established as research projects to test the teaching effi- 
ciency of groups of components. Few school systems or research centers 
have records of the details of components of programs, their costs, or 
the results. The aim of authorities involved in compensatory education 
has been to teach rather than to keep such records, and their attitude 
tends to be that the data necessary for cost-effectiveness studies 
probably do not exist, but if they do, many man/days would be required 
to extract thun. 

Selected programs could be assisted to establish and maintain the 
types of records necessary for cost-effectiveness studies. Experience 
in this survey has shown that there are some programs which would toler- 
ate such assistance, and others which would welcome it. The immediate 
purpose would be to build into these programs tests and observations, 
records and accounts in such a way that some analysis would become 
feasible. Since the programs themselves would have access to the raw 
data from the "dials and gauges" installed, it would be generally to 
their advantage to cooperate. Those who assisted, the programs to make 
these changes would be consultants on the structuring of recording and 
evaluation, rather than being evaluators. Their services would assist 
in the modification of programs in ways which would not hinder opera- 
tions, but would improve eva 1 nation. 

Since such a scheme would be accepted only by authorities who 
wished it, it could not be construed as a plan to introduce more govern- 
mental control. To A.I.R., it represents the best chance of moving 
towards cost-effectiveness studies in this field. 

To conclude, seme further statements on cost-effectiveness should 
be quoted from Prest and Turvey: 

An important advantage of a cost-benefit study is that it 
forces those responsible to quantify costs benefits as 
far as possible rather than rest content with vague quali- 
tative judgments or personal hunches* This is obviously a 
good thin g in itself; some information is always better * h*n 
none. Furthermore, quantification and evaluation of bene- 
fits, however rough, does give some sort of clue to the 
charges which consumers are willing to pay. It may well be 
a salutary check on the biases likely to creep into esti- 
mated costs and benefits by enthusiastic advocates of parti* 
cular projects, that, wherever technically feasible, some 
charges should be imposed* The discipline of the market- 
place is so easily and so readily forgotten in these situ- 
ations that some empirical evidence about benefit projec- 
tions is highly necessary. Insistence on some charging 
process may therefore be a sensible antidote to wilder ex- 
cesses of particular lobbies. We must remember , however, 
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that the extent to which the authorities impose these 
charges brings in distributional as well as efficiency 
calculations. 

Even if cost-benefit analysis cannot give the right 
answers, it can sometimes play the purely negative role 
of screening projects and rejecting those answers which 
are obviously less promising . . . 

The case for using cost-benefit analysis is strength- 
ened, not weakened, if its limitations are openly recog- 
nized and indeed emphasized. ... It is no good expecting 
thos : fields in which benefits are widely diffused, and 
in wh’ch there are manifest divergences between account- 
ing and economic costs or benefits, to be as cultivable 
as others. Nor is it realistic to expect that compari- 
sons between projects in entirely different branches of 
economic activity are likely to be as meaningful or 
fruitful as those between projects in the same branch. 

The technique is more useful in the public-utility area 
than in the social-services area of government. Compari- 
sons between, say, different road projects are more help- 
ful than those between, say, road and water projects; and 
both these are likely to be more helpful than applications 
in the fields of education, health, research, and so on. . 
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1966. 

National Conference on Educational Objectives for the Culturally Dis- 
advantaged. Education for the culturally disadvantaged . 

Little Rock, Arkansas: South Central Region Educational Labora- 

tory. (Undated.) 

National Council of Teachers of English, Task Force on Teaching 

English to the Disadvantaged. language program for the dis- 
advantaged: The report of the NCTE Task Force . Chicago: NCTE, 

1965. 

National Education Association, American Association of School Admin- 
istrators and Research Division. School programs for the dis- 
advantaged . (Educational Research Service Circular No. 1) 
Washington, D. C.: NEA, 1965. 

National Society for the Study of Education, Sixty-seventh Yearbook , 
Part I — Metropolitanism: Its challenge to education . Chicago : 

University of Chicago Press, 1968. 
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New York State Education Department reports on Project ABLE as follows: 

(New York State Education Department) Bienenstok, T., & Sayres, W. C. 
Project ABLE: An appraisal . Albany: University of the State 

of New York, State Education Department, Division of Research, 1964. 

(New York State Education Department) Project ABLE . Albany: University 

of the State of New York, State Education Department, Division of 
Pupil Personnel Services, 1968. • 

University of the State of New York, State Education Department, Bureau 
of Guidance. Project ABLE: The first year . Albany: The Bureau, 

1963. 

University of the State of New York, State Education Department, Bureau 
of Guidance. Helping educationally disadvantaged c hildren: The 

second year of Project ABLE . Albany: The Bureau. (Undated.) 

University of the State of New York, State Education Department, Division 
of Pupil Personnel Services. Project ABLE: A manua l for school 

districts interested in developing a program . Albany: The Division, 

1968. 

University of the State of New York, State Education Department, Division 
of Pupil Personnel Services. Local schools* emphases within ABLE 
projects: School year 1967—68 . Albany: The Division. (Undated.) 

New York State Education Department reports on Prekindergarten Programs : 

(Hew York State Education Department) DiLorenzo, L. T., & Salter, R. 

An evaluative study of pre-kindergarten programs for educa tionally 
disadvantaged children. 1965-1967 . Albany: University of the 

State of New York, State Education Department, Office of Research 
and Evaluation, 1967. 

(Hew York State Education Department) DiLorenzo, L. T., & Salter, R. 

evaluative study of prekindergarten programs for educationally 
disadva ntage d children: Follow-up and replication. (Paper pre- 

sented at a meeting of the American Educational Research Associa- 
tion in 1968.) Albany: University of the State of New York, 

State Education Department, Office of Research and Evaluation, 1968. 

(New York State Education Department) Evaluative study of prekindergar- 
ten programs for educationally disadvantaged childre n: Test results. 

Albany : University of the State of New York, State Education Depart- 

ment, Office of Research and Evaluation, November 1967. 
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New York State Education Department reports on the STEP Program: 

(New York State Education Department) Bienenstok, T., & Sayres, W. 

STEP School to Employment Program: An appraisal . Albany : 

University of the State of New York, State Education Depart- 
ment, Division of Research, 1964. 

(New York State Education Department) Developing work-study j>ro- 
grams for potential drop-outs: A manual. STEP School to 

Employment Program . Albany: University of the State of New 

York, State Department of Education, Bureau of Guidance, 1965. 

(New' York State Education Department) Moore, J. W. School to 

Employment Program (STEP ). Albany: University of the Sta-e 

of New York, State Education Department, Bureau of Guidance, 

April 1963. 

(New York State Education Department) Moore, J. W. STEP, The School 
to Employment Program: Cases in point . Albany : University 

of the State of New York, State Education Department, Bureau 
of Guidance, 1964. 

University of the State of New York, State Education Department, 

Bureau of Guidance. School to Employment Program (STEP) : 

Se co n d annual report, 1962—1963 . Albany: The Department, 1963. 

University of the State of New York, State Education Department, 

Division of Pupil Personnel Services. STEP School to Employment 
Progr*» : A «a"«al of information and instruction for preparing 

an application for a STEP program . Albany: The Department, 1968. 

New York University, School of Education, Institute for Developmental 
Studies . Selected bibliography: Institute materials — related 

bibliographical references . New York: The Institute, April 

1967. 

Northwest Regional Educational Lab. Improving education for cultur- 
ally different children: Program 200 . Portland, Oregon: The 

Lab. (Undated.) 

Passow, A. H. (Ed.) Education in depressed areas . New York: Colum- 

bia University Teachers College, 1963. 

Passow, A. H. (Ed.) Education of the disadvantaged: A book of read- 
ings . New York: Holt, Rinehart, and Winston, 1967. 

Peerboom, P. Report on Summer 1967 site visits to pre-Title I com- 
pensatory education programs . Washington, D. C.: U. S. Depart- 

ment of Health, Education, and Welfare, Office of Education, 
Bureau of Elementary and Secondary Education, Division of Com- 
pensatory Education, 1967. 
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(P enns ylvania Department of Public Instruction) Preschool and Primary 
Education Project annual progress reports: 1963-64, 1964-65 , 

1965-66. and 1966-67 . Harrisburg: Pennsylvania Department of 

Public Instruction, the Project, 1964, 1965, 1966, and 1967, 
respectively. 

Pines , M. Revolution in learning: The years from birth to six . 

New York: Harper 6 Row, 1967. 

Potts, A. M. Developing curriculum for Indian children . Alamosa, 
'Colorado: Adams State College, 1964. 

Potts, A. M. Knowing and educating the disadvantaged: An annotated 

bibliography . Alamosa, Colorado: Adams State College, The Cen- 

ter for Cultural Studies, 1965. 

Reiff, D. G. , & Pere, J. The language situation in Project Head 

Start centers . (OEO-932) Washington, D. C.: Office of Economic 

Opportunity, Office of Research and Evaluation, 1965. 

Rpyearch in Education , 1966- , 1- (A monthly abstract journal pre- 
pared by the Educational Resources Information Center (ERIC) 
and published by the U. S. Department of Health, Education, and 
Welfare, Office of Education, Bureau of Research, Division of 
Information Technology and Dissemination.) 

Review of Educational Research. 1965, 35(5). (An issue devoted to 
education for "socially disadvantaged" children.) 

Savitsky, C. Dropouts and holding power. The Clearing House , 1963, 

38, 89-92. 

Savitsky , C. Social theory advances on the disadvantaged. High 
Point , 1964, 46 , 54-62. 

School and Society , 1968, 96(2306). (A special issue devoted to 
"the disadvantaged and education.") 

Suchman, E. A. Evaluative research: Principles and practice in pub- 
lic service and social action programs . Hew York: Russel Sage 

Foundation, 1967. 

Texas Education Agency. Preschool Instructional Program for non-English 
speaking children . (Bulletin No. 642) Austin: The Agency, 1964. 

Texas Education Agency. The Texas Project for Migrant Children: An 

evaluation of first year operation of pilot project in five school 
districts. Austin: The Agency, 1964. 
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Thompson, D. C. Evaluation as a factor in planning programs for the 
culturally disadvantaged Journal of Negro Education , 1964, 
33(3), 333-340. 

Thompson, H. (Ed.) A digest of information on the education of 
Indians. Indian Education , 1964, No. 399. 

United States Department of Health, Education, and Welfare, Office 
of Education. Programs for the educationally disadvantaged . 
Washington, D. C. : U. S. Government Printing Office, 1963. 

United States Department of Health, Education, and Welfare, Office 
of Education. A chance for a change: New school programs for 

the disadvantaged . (0E-35084) Washington, D. C.: U. S. Govern- 

ment Printing Office. (Undated.) 

United States Department of Health, Education, and Welfare, Office 
of Education. Special report: Summer projects . (0E-37008) 

Washington, D. C. : U. S. Government Printing Office. (Undated.) 

University of the State of New York, State Education Department. 
Elementary and Secondary Education Act of 1965 — Title 1: The 

New York State annual evaluation report for 1965-66 fiscal year . 
Albany, New York: The Department, 1966. 

University of the State of New York, State Education Department, 

Office of the Coordinator-Title 1, ESEA. Mathematics education 
programs funded under Title I Elementary and Secondary Education 
Act of 1965 . Albany, New York: The Department. (Undated.) 

University of the State of New York, State Education Department, 

Office of the Coordinator-Title I, ESEA. One hundred selected 
projects. Albany, New York: The Department. (Undated.) 

University of Wisconsin, Center for Studies in Vocational and Techni- 
cal Education. Bibliography XIV, Addendum 1: Disadvantaged 

groups . Madison: The Center, October-Dec ember 1966. 

University of Wisconsin, Center for Studies in Vocational and Techni- 
cal Education. Bibliography XIV, Addendum 4: Disadvantaged 

groups . Madison: The Center, May-August 1967. 

Urban Review , 1966(?)- , 1- (A periodical devoted to the disadvan- 
taged and published by the Center for Urban Education, New York, 
New York.) 

r 

Watson, G. (Ed.) No room at the bottom: Automation and the reluctant 

learner . Washington, D. C. : National Education Association, 

1963. 

Weikart, D. P. Preschool programs: Preliminary findings. Journal 

of Special Education , 1967, 1(2), 163-181. 
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Western States Small Schools Project: Summer workshop . Nevada: 

University of Nevada, Department of Education, 1963. 

Western States Small Schools Project: University of Nevada . Nevada : 

The University, Department of Education, 1964. 

Wilkerson, D. A. Bibliography on the education of the socially dis- 
. advantaged. Journal of Negro Education , 1964, J3, 358-366. 

Yeshiva University. Preschool education: A selected bibliography 

on the disadvantaged . New York: The University, ERIC Clearing 

House, Harch 1967. 

* Yeshiva University, Ferkauf Graduate School of Education. Bibliogra- 
phy on the education of socially disadvantaged children and 
youth . New York:- Information Retrieval Center on the Disadvan- 
taged (IRCD), the University. (Undated.) 

Yeshiva University, Ferkauf Graduate School of Humanities and Social 
Sciences. The education of teachers of the disadvantaged: A 

selected bibliography . • New York: Information Retrieval Center 

on the Disadvantaged (IRCD), the University, March 1967. 

. Sources for Programs Studied but not Described 

Note: Programs are listed alphabetically within a given state. 

•If more than one project is known by the same name within a state, 
such projects are ordered by city. These sources refer to documents 
collected by A.I.R. during the study, and do not include references 
examined at ERIC and other locations. 

ARIZONA 

Cooperative Project 

. . f 

Arizona Research and Development Center, Early Childhood Education 
Laboratory. Why the Project . Tucson: The Laboratory, 1968. 

Coxon, M. L. The construction of a tine concept . Tucson: Arizona 

Research and Development Center, Early Childhood Education 
Laboratory, 1968. 

Henderson, R. , & Wetzel, R. J. The Tucson early education model : 
Summar^ of a model for Follow Through Program development and 
■' implementation . (Cooperative Research Project, Tucson School 

District No. 1, University of Arizona) Tucson: Arizona Research 
and Development Center, Early Childhood Education Laboratory, 
1968. 

► 

* >; • - * 

Language Program 

Stout, 1. W. , & Langdon, G. The use of toys in teaching English to 

non-English speakir? children . Tempe : Arizona State University, 

College of Education, 1964. 
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Reading Program 

Fearn, L. A dual directional approach to Initial reading Instruction : 

A pilot study . Lukachukai, Arizona: Navajo Demonstration School* 

1966. 

CALIFORNIA 

Collections of Evaluation Reports 

Eureka City Schools. Report of ESEA Title I projects, 1967-68 : 

Room to Grow-Phase II, Room to Explor e-Phase II, Room to Study- 
Phase I, and Room to Continue-Phase III . Eureka, California: 

The Schools, 1968. 

Eureka City Schools. Report of ESEA Title I Educational Skills Cen- 
ter. 1966: A chance to continue . Eureka, California: The 

Schools . (Undated . ) 

Eureka City Schools. Report of ESEA Title I projects, 1966-67 : 

Room to Grow, Room to Explore, Room to Continue . Eureka, 

California: The Schools. (Undated.) 

(Garden Grove Unified School District) Legree, H. F., Anton, T. , & Jordon, 
G. C. Compensatory education: An evaluation, 1966-67 . Garden 

Grove, California: The District, August 1967. 

Los Angeles City School Districts. ESEA Title I projects evaluation 
reports: Spring semester, 1966 . Los Angeles: The Schools, 

August 1966. 

Los Angeles City School Districts. ESEA Title I project: The third 

year, 1967-1968 . Los Angeles: The Schools, August 1967. 

Los Angeles City School Districts. ESEA Title I evaluation reports : 
September 1966 through August 1967 . Los Angeles: The Schools, 

September 1967. 2 vol. 

Los Angeles City School Districts. ESEA Title I project: The third 

year, 1967-1968: Summary . Los Angeles: The Schools, October 

1967. 

Los Angeles City School Districts. Progress reports: ESEA-I, EO A, 

and ESEA-III . Los Angeles: The Schools, February 1968. 

Los Angeles City School Districts, Division of Secondary Education, 

Office of Specially Funded Programs. ESEA Title I project su m- 
mary: Spring, 1966 . (Re Student Achievement Centers and rela- 
ted projects.) Los Angeles: The Division, March 1966. 

Los Angeles City School Districts, Division of Secondary Education, 

Office of Specially Funded Programs. Summary, 1966-67: Elemen- 

tary and Secondary Education Act, Title I . (Re Student Achieve- 
ment Centers and related projects.) Los Angeles: The Division, 

October 1966. 
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Los Angeles City School Districts, Office of Research and Development. 

ESEA Title I evaluation forms and instruments: September 1966 

through August 1967 . Los Angeles: The Schools, November 1967. 

Los Angeles City School Districts, Office of Research and Development. 

ESEA Title I projects: Summaries of evaluation report s, 1966- 

1967 . Los Angeles: The Schools, November 1967. 

Modesto City Schools (E. M. Azevedo). Modesto City Schools comprehen- 
sive program of compensatory education . Modesto, California: 

The Schools, It 67. 

Oakland Public Schools, Interagency Project. Summary digest of the Inter - 
agency School Project. Oakland, California: The Schools, 1964. 

Oakland Public Schools , Research Department. Evaluation report: ESEA 

program of compensatory education . Oakland, California: The 

Schools, August 1967. 2 vol. 

Oakland Public Schools, Research Department (A. W. Badal) . ABSTRACT : 
Evaluation report — ESEA program of compensatory education . 

Oakland, California: The Department, September 1967. 

Oakland Unified School District, Board of Education. ESEA program of 
compensatory education. 1966~1967 . Oakland, California: The 

District, 1966. 

Oakland Unified School District, Board of Education. ESEA program of 
compgtt«atory education, 1967~1968 . Oakland , California : The 

District, 1967. 

San Diego Unified School District, Testing Services Department, ESEA 
Evaluation Unit. Evaluation report of compensatory education 
program. 1966-67: Part I-Secondary program (including elemen- 
tary and secondary baseline data ) . San Diego , California : The 

Department, November 1967. 

San Diego Unified School District, Testing Services Department, ESEA 
Evaluation Unit. Evaluation report of compensatory education 
program. 1966-67: Prt II-Elementary program . San Diego, Cali- 
fornia: The Department, November 1967. 

(San Francisco Unified School District) Sorenson, P. H., & Thomas, T. C. 
Evaluation of the compensatory education progra m of the San Fran- 
cisco Unified School D istrict, 1966-1967: Detai led findings. 

Menlo Park, California: Stanford Research Institute, 1967. 

(San Francisco Unified School District) Sorensen, P. H. , & Thomas, T. C. 
gimwnar y evaluation of the compensatory education pro gram of the 
San Francisco Unified School District, 1966-1967 . Menlo Park, 
California: Stanford Research Institute, 1967. 
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(San Francisco Unified School District) Sorensen, P. H. , Thomas, 

T, C., & Goldupp, 0. Appendixes to: Evaluation of the compen- 

satory education program of the San Francisco Unified School 
District, 1966-1967 , Menlo Park, California: Stanford Research 

Institute, 1967. 

Santa Rosa City Schools . Annual evaluation (of ESEA-Title I program ) . 
Santa Rosa, California: The Schools, 1967. 

Specific Projects 

Advanced Math Program for Elementary School Children 

Rupley, W. H. Teaching of advanced math concepts to culturally dis- 
advantaged elementary school children . Berkeley, California: 
University of California, 1966. 

Back to School Project 

Los Angeles City Schools, Division of Secondary Education, Evaluation 
and Research Section. Back to School Project: A report of an 

experimental program to aid dropouts . Los Angeles: The Division. 

(Undated. ) 

Basic Skills Development Project 

Wilson, J. A. R. Annual evaluation report: Basic Skills Development 

Project . Santa Barbara, California: Santa Barbara High School 

District, 1967. 

Wilson, J. A. R. Summer school evaluation retort: Basic Skills 

Development Project . Santa Barbara, California: Santa Barbara 

High School District, 1967. 

Compensatory Education Program (Berkeley) 

Jonsson, H. A. Effectiveness of ESEA Title I activities in the 

Berkeley Unified School District: A short summary of evaluation 

for the 1966-67 project year . Berkeley, California: The Dis- 

trict, 1967. 

Jonsson, H. A. Report of evaluation of ESEA Title I compensatory 
education activities for 1966-67 . Berkeley, California: 

Berkeley Unified School District, 1967. 

Compensatory Education Program (San Diego) 

San Diego City Schools. Pilot project in compensatory education : 

A report to the State Adv i sory Committee on Compensatory Educa- 
tion . San Diego, California: The Schools, January 1965. 



Co gg ensatory Education Program (San Fra n c i sco) 

San Francisco Unified School District, Elementary Division. Superin- 
tendent *s Compensatory Education Progr am (in the elenentarx 
schools) . San Francisco: The Division, June 1965. 

San Francisco Unified School District, Elementary Division. Study 
of pupil reading records of elenentary compensatory children. 

Sa n Francisco: The Division, 1967. 

San Francisco Unified School District, Office of Compensatory Educa- 
tion. State Compensatory Program . San Francisco: The Office, 

June 1965. 

San Francisco Unified School District, Office of Compensatory Educa- 
tion. Compensatory education . San Fra n c i sco : The Office, 196 . 

San Francisco Unified School District, Senior High Division. Superin- 
tendent^ Compensatory Education Program (in the senior high 
schools) . San Francisco: The Division, June 1965. 

Counseling Program 

Oakland Interagency Project. Report on the evaluation of the Inter- 
agency School Project Counseling Program at el ementary and junior 
schooll Oakland, California: The Project, 1964. 

Cultural and Communications Project 

Williams, A., Pettingal, J. F., & Mueller, 0. San Gabriel Cultural 
ynH Communications Project . San Gabriel, California: San 

Gabriel School District, July 1967. 

Dr«« Demonstration Project 

DeCecco, J. P., & VanCovering, H. Progress re port: Evaluation_qj 

the Drama Demonstration Project, in-school pr ogram and out-of- 
school program . San Francisco: San Francisco Unified School 

District, 1965. 

English as a Second Language (El Centro) 

Imperial County Schools. Teaching Englis h as a second language__to 
pupils of foreign bom. Mexican heritage: Lesson plans. 

El Centro, California: The Schools, 1963. 

PnpUch as a Second Language (Los Angeles) 

Los Angeles City School Districts, Division of Elementary Education, 
Office of Specially Funded Programs. ESEA Titl e I Summer 1968 
English as a Second Language Program . Los Angeles: The Office, 

April 1968. 
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English as a Second LangiMg* (San Diego) 

Aschbrenner, A., A Jorgenson, G. W. Guidelines for t^rhinp EijgHvh 
as a second language in secondary schools . (San Diego Project- 
ESEA) San Diego, California: San Diego City Schools, 1966. 

Baa d Start Program (Fullerton) 

Slaven, J. J. Montessori Head Start. Audiovisual Instruction. 1966. 
11(7), 547-549. 

Head Start Progras (Garden Grove) 

Legree, H. F. Compensatory education: Project Head Start— An evalua- 
tion, Su— er 1967 . Garden Grove, California: Garden Grove 

Unified School District, 1967. 

Head Start Program (Oakland) 

Oa kl a n d Public Schools, Research Department. Report of evaluation (of) 
Head Start Progras, Sumner 1966 . (Report Ho. 3) Oakland, Califor- 
nia: The Schools, 1967(?) 

Oakland Public Schools, Research Department. Report of evaluation (of) 
Head Start Program, Suamer 1967. (Report Ho. 4) Oakland, Califor- 
nia: The Schools, 1968 (?) 

Home Visitation Proya* 

Ravenswood City School District. Hone visitation policy . Palo Alto, 
California: The District, June 1967. 

Villa, A. J. Evaluations of a compensatory program of visits to the 
hones of pupils of Belle Haven School in the Ravenswood City 
School District under the McAteer Act . Palo Alto, California: 

The District, August 1965. 

L a n gu a ge Awareness Project 

Oakland Interagency Project. Report of evaluation of kindergarten 
Language Awareness Program at Stonehurst Elementary School . 

Oakland, California: The Project, 1964. 

Langu age' Development Program 

Oakland Interagency Project. Report of evaluation of third and fourth 
grade Language Development Program . Oakland, California: The 

Project, 1964(?) 

Oakland Public Schools, Office of the Director of Elementary Education. 
Multi-media approach to reading and language development . Oakland , 
California: The Office, February 1966. 



Oakland Public Schools, Office of Special Urban Educational Services. 

Sugary of evaluation report; Stonehurst Primary Languag e Develop- 
■ent Program. Oakland, California: The Office, February 1966. 



Oakland Public Schools, Office of Special Urban Educational Services. 

Abstract of SB-28 Demonstration Project: Compensating for cu ltural 

and linguistic interference in reading . Oakland, California: The 

Office, February 1967. 



L a n g u age Enrichment Program 

Oakland Interagency Project. Report of evaluation of Language Enrich - 
ment Program a t Stonehurst Child Care Center . Oakland , Califor- 
nia: The Project, 1964. 

Language Training Program 

Bean, J. P. A specific langua ge training program for pre school Mexican- 
American children. Paper presented at the Western Psychological 
Association meeting at San Diego, California, March 1968. San Jose, 
California: San Jose State College, 1968. 

Mathematics Curriculum Project (SMSG) 

Chinn, W. G. , & Suomerf ield , J. 0. The Special Curriculum Project . 
1965-66: Pilot program on mathematics learning of culturally * 

disadvantaged p rimary school children . (SMSG Report No. 4) 

Stanford, California: Stanford University, 1967. 

Leiderman, G. F., Chinn, W. G. , & Dunkley, M. F. The Special Curriculum 
Project: Pil ot program on mathematics leanring of culturally dis - 

advantaged primary school children . (SMSG Report No. 2) Stanford 
California: Stanford University, 1966. 

Oral Language Development Project 

Oakland Public Schools, Research Department. Evaluation of Willow Manor 

Oral Lang uage Development Project . Oakland, California: The 

Department, August 1964. 



Preschool Group Experience Progr am 

Vance, B. J. FLnal report: T he effect of preschool group experience on 

various language a nd social skills in disadvantaged children (nf 
fremont Unified Schoo l District) . Stanford, California: Stanford 

University, 1967. (Republished: Washington, D. C. , U. S. Department 
of Health, Education, and Welfare, Office of Education, Bureau of 
Research, 1967.) 
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Preschool Language Project 



Stern, C. The Preschool Language Project: A report of the first 

year's work . Los Angeles: University of California at Los 

Angeles, 1966. 

Preschool Program 

Oa k la n d Public Schools, Research Department. Report of evaluation of 
Oakland Public Schools Assembly Bill 1331 Preschool Program , 
1966-67 . (Research Report No. 5) Oakland, California: The 

Schools, 1968. 

Oakland Public Schools, Research Department. Preliminary report of 
evaluation of Preschool Program at Clawson. Cole, and Stone- 
hurst Schools— 1964-65 . (Report No. 12) Oakland, California: 

The Schools. (Undated.) 

Preschool II and IV Program 

Los Angeles City School Districts, Division of Elementary Education. 
The pre-school teacher aide: Ten tative form (for) teacher use. 

Los Angeles: The Division, December 1966. 

Los Angeles City School Districts, Division of Elementary Education. 
Suggestions for implementation of broad goals: Pre-school . 

Tentative form (for) teacher's use . Los Angeles: The Division, 

December 1966. 

Los Angeles City School Districts, Division of Elementary Education. 
Parent involvement in the pre-school program . Los Angeles : 

The Division, January 1967. 

Los Angeles City School Districts, Division of Elementary Education. 
Summary of pre-school experiences . Los Angeles: The Division, 

January 1967. 

Los Angeles City School Districts, Division of Elementary Education. 
Developmental tasks . Los Angeles: The Division, August 1967. 

Los Angeles City School Districts, Division of Elementary Education. 
Suggested two-day outline for pre-school . Los Angeles: The 

Division, September 1967. 

Los Angeles City School Districts, Division of Elementary Education. 
Suggested two-week outline for pre-school . Los Angeles: The 

Division, September 1967. 
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Reading Articulation Program 

Articulation of Reading Committee. The Elementa ry Reading Articula- 
-pion Pro g ram: — A languag e arts program with emphasis on reading. 
Fresno, California: Fresno City Unified School District, 1967. 

Reading Augmentation Project 

Dean, L. M. , Nichols, G. C. , Zuckerman, D. G. , & Gallaher, P. J. Evalua— 
report. Au gmented reading implementation project, 1967 . 

Pomona, California: Pomona Unified School District, 1967. 

Reading Program for Mexican-American Children 

Amsden, C. A reading program for Mexican-American children: Fi rst 

interim report. Los Angeles: California State College, 1966. 

Amsden, C. Second interim repor t: A reading program for Mexican- 

American children . Washington, D. C.: U. S. Department of 

Health, Education, and Welfare, Office of Education, Bureau of 
Research, 1968. 

Reading Specialist Program 

Los Angeles City School Districts, Division of Elementary Education. 
Primary reading spe cialists* evaluation of in-service, August 15— 

26, 1966: Suggestion s of further in-service in areas and in the 

district . Los Angeles : The Division, 1966. "" 

Los Angeles City School Districts, Division of Elementary Education, 
Office of Specially Funded Programs. Developmental skills (read- 
ing). Los Angeles: The Office, October 1967. 

Los Angeles City School Districts, Division of Elementary Education, 
Office of Specially Funded Programs. Suggestions for meeting 
some specific r eading needs of children through diagnostic teach- 
ip 8 » Los Angeles : The Office, October 1967. 

Los Angeles City School Districts, Division of Elementary Education, 
Office of Specially Funded Programs. ESEA reading specialists * 
^valua tion of books they have used— which are being considered 
for state adoption. Los Angeles: The Office, February 1968. 

Los Angeles City School Districts, Division of Elementary Education, 

Office of Specially Funded Programs. Summ ary of Reading Specialist 

Program, S eptember 1967 through June 1968 . Los Angeles: The 

Office, 1968. 

Los Angeles City School Districts, Division of Instructional Services, 
Instructional Aids and Services Branch, Library Section. ESEA 
Title I 1965 (ma terials lists for reading specialists, corrective 
and remedi al reading, non-English speaking pupils, early child- 
hood and kindergarten). Los Angeles: The Section. (Undated.) 
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Remedial Reading Program 

Colwell, H. P. Redondo B each City School District compensatory educa- 
tion ESEA Title I ev aluation, 1966-67 . (Remedial reading was the 
activlt y*> Redondo Beach, California: The School District, 

1967 • 

Colwell, H. P. Redondo Beach City School District compensatory educa- 
— — on ESEA Title X evalu a tion, 1967~1968 . (Remedial reading was 

the basic activity.) Redondo Beach, California: The School 

District, 1968. 

School— Conmunity Improvement Progr am 

Pivnick, I. School-Community Improvement Program: School year 1962-63 . 

San Francisco! San Francisco Unified School District, June 1963. 

Pivnick, I. School-Community Improvement Program: Final report , 

1961-64. ban Francisco: San Francisco Unified School District 

1964. 

Special Instructional Programa 

Oakland Interagency Project. Report of evaluation of Special Instruc- 
tional Programs at Madison Jr. High School: School years 1962^63 

and 1963-64. Oakland; California: The Project, 1964. 

Study Centers Program 

Oakland Interagency Project. Report of evaluation of five selected 

study center s: School year 1963-64 . Oakland, California: The 

Project, 1964. 

Study Skills Center Program 

Los Angeles City School Districts, Division of Secondary Education 

and Division of College and Adult Education, Office of Specially 
Funded Programs. Specifications for SFP Study Skills Center : 

School y ear 1966-67 . Los Angeles: The Office, 1966 (?) 

Summer Guidance School Project 

Hickman, R. C. The dropouts did come back: A special Summer Guidance 

School Program. California Education . 1964, 11, 3-9. 

Summer School Program 

Laliberte, R. A. Evaluation of the 1965 secondary Smnmer School 

Program. Oakland, California: Oakland Unified School District, 

1966. 

Tutoring Program 

Oakland Interagency Project. Effect of tutoring on performance and 
motivation rati ngs in secondary school students . Oakland , Cali- 
fornia: The Project, 1964. 
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COLORADO 



Specific Projects 

Individualized Reading Program 

Nichols , M. Individualized Reading Program for elementary grades * 
Denver: Colorado State Department of Education, Western 

States Small Schools Project, 1964. 

New Nursery School Program 

Nimnicht, G. P. New Nursery School . Greeley: Colorado State 

College, January 1968. 

Nimnicht, G. P., et al. A project in nursery school education for 
environmentally deprived Spanish-American children: First year 

progress report . Greeley: Colorado State College, 1966. 

Nimnicht, G. , McAfee, 0., & Meier, J. Inservice education for 
Headstart teachers and aides: For field testine by The Far 

, West Laboratory for Educational Research and Development . 
(Training units I, III, IV, VI, VII, VIII, X, XI, and XVI 
with Conclusion) Greeley: Colorado State College, New Nursery 

School, 1968(?) 



CONNECTICUT 

Collection of Evaluation Reports 

(Hartford Board of Education, Research Evaluation) Nearine, R. J. 
Where the action is, 1966-1967: An evaluation . Hartford, 

Connecticut: Hartford Public Schools, 1967. 



Specific Projects 
Catch Up Project 

Nearine, R. J. Where the action is. 1966—1967: An evaluation (of 

Project Catch Up) . Hartford, Connecticut: Hartford Board of 

Education, 1967. 
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Dropout Prevention Program 

Walden, J. Report for NSIP No. 3s Dropout Prevention Program . 

South Norwalk, Connecticut: Benjamin Franklin Jr. High, 1968. 

English as a Second Language 

Nearine, R. J. Where the action is, 1966-1967: An evaluation (of 

English as a Second Language) . Hartford, Connecticut: Hartford 

Board of Education, 1967. 

Higher Horizons 100 Program 

Hartford Board of Education, Research Evaluation Department. Some 
comparisons following the first marking period of the 1966-67 
school year . Hartford, Connecticut: The Department, 1967. 

Hartford Board of Education, Research and Publication Department. 
Intensive language instruction, experiential development and 
guidance report: Evaluation 1965-1966 . Hartford, Connecticut: 

The Department, 1966. 

Nearine, R. J. Where the action is, 1966~1967: An evaluation (of 

the Higher Horizons 100 Program) . Hartford, Connecticut: Hart- 

ford Board of Education, 1967(7) 

Nearine, R. J. Patterns for progress: An evaluation, 1967-1968 (of 

the Higher Horizons 100 Program) . Hartford, Connecticut: Hart- 

ford Board of Education, 1968. 

Remedial Learning Centers Project 

Nearine, R. J. Where the action is. 1966—1967: An evaluation (of the 

Remedial Learning Centers Project) . Hartford, Connecticut: 
Hartford Board of Education, 1967. 

Work-Study Program 

Anderson, P. 0.(?) Ninth grade Work-Study statistics: An interpretive 

summary. New Haven, Connecticut: New Haven Public Schools, 1966. 

New Haven Public Schools (7) Ninth grade Work-Study Program. Basset 

Junior H igh School . New Haven, Connecticut: The Schools(?) . 

1963(7) * 

New Haven Public Schools. Junior high school Work-Experience Educa- 
tion Program. New Haven, Connecticut: The Schools, June 1964. 

Zaorski, R. C. Ninth-grade Work Study Program . New Haven, Connecticut: 
New Haven Public Schools, 1965. 
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DISTRICT OF COLUMBIA 

9 

Collections of Evaluation Reports 

Dailey , J. T., & Neyman, C. A., Jr. Evaluation of ESEA Title I pro- 
grams for t he District of Columbia, 1966 and 1967: S»™ary 

report. Washington, D. C. : George Washington University, 

Educational Research Department, 1967. 

District of Columbia Public Schools, Model School Division. 

Model School Division: A report to the community . Washington, 

D. C.: The Division, February 1968. 

Specific Projects 

Guidance Project 

Columbia Public Schools, MacFarland-Roosevelt Demonstra- 
tion Guidance Project Staff. Final report of: 1959-1965 . 

Washington, D. C. : The Schools, 1965(?) 

Language Arts Project 

Dailey, J. T., & Neyman, C. A., Jr. An evaluation of the Language 
Arts Program of the District of Columbia: Final rcpuil . Wash- 
ington. D- C„ : Ceoige Washington University, 1965. 

Kornhauser, L. H. The Language Arts Project. 1962— 63 i For members 

of the Board of Education of the District of Columbia^ New York: 
Columbia University Teachers College, 1963. 

Kornhauser, L. H. The Language Arts Project report. 1963-1964: For 

me mbers of the Board of Education of the District of Columbia . 
Washington, D. C. : The Project, 1964. 

Kornhauser, L. H. A general evaluation of the Language Arts Program 
for culturally disadvantaged pupils, April 1967 . Washington, 

D. C. : Language Arts Project, 1967. 

Progressive Choice Reading Project 

Woolman, M. Evaluations of the progressive choice reading method . 
Washington, D. C. : Institute of Educational Research, 1964. 

"Roundabout" Television Project 

Mukerji, R. A national demonstration project utilizing televised 
materials for the formal education of culturally disadvantaged 
preschool children . Washington, D. C.: U. S. Department of 

Health, Education, and Welfare, Office of Education, Bureau of 
Research, 1966. 

School to Aid Youth Program (ST AY) 

New worlds unfold at Spingarn STAY High School. Fall 1967-68 . Washing 
ton, D. C.: The School. (Undated.) 
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FLORIDA 



Learning to Learn Program 

Sprigle, H., et al. A fresh approach to early childhood education 
and a study of its effectiveness: Learning to Learn Program . 

Jacksonville, Florida: Psychological Clinic and Research 

Center, 1968(?) 

Programmed Reading Project 

Malpass, L. F. Programmed reading instruction for culturally deprived 
slow learners. Tampa, Florida: McDonald Training Center Founda- 

tion, August 1966. 

Reading Readiness Program 

Spache, G. D. A study of a longitudinal first grade Reading Readiness 
rogram. Tallahassee, Florida: Florida State, Department of 

Education, Division of Instruction, 1965. 

Summer Reading and Enrichment Program 

Shaw, L. M. Summer reading and enrichment for the educationally 

1966 (?) d * Gains eville, Florida: Board of Public Instruction, 



ILLINOIS 

Collection of Evaluation Reports 

Willis, B. C. Compensatory education in the Ch icago Public Schools: 
Study report No. 4 . Chicago: The Schools, 1964. 

Specific Projects 

After School Study Centers Program 

Janowitz, G. After-School Study Centers: Volunteer work in reading . 

Chicago: Commission on Human Relations, Mayor* s Committee on 

New Residents, 1964. 

Janowitz, G. Helping hands: Volunteer work in educa tion. Chicago: 

University of Chicago Press, 1965. 

Great Cities School Improvement Project 

Bloom, S. School report: The Great Cities School Improvement Project . 

Chicago: University of Chicago, Conference on Compensatory Educa- 

tion, 1964. 
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Reading Readiness Laboratory — 

Strodtbeck, F. L. Progress report: The Reading Readiness Laboratory . 

Chicago: University of Chicago, Social Psychology Laboratory, 1963. 

Strodtbeck, F. L. The Reading Readiness Nursery: Short term social 

intervention t echnique . Chicago: University of Chicago, Social 

Psychology Laboratory, 1964. 

School-University Education Project 

Us dan , H. D.<, & Bertolaet, F. Development of a school— uni versity pro— 
gram fo r the preservice education of teachers for th e disadvantaged 
through t eacher education centers . Evanston, Illinois (?) : School 

Improvement Program, Research Council of the Great Cities(?), 1965. 

UBdan, M. D. , & Bertolaet, F. Teachers for the disadvantaged . Chicago: 
Follett, 1966. 

Special Summer Schools^ Program (Elementary) 

Chicago Board of Education. The story of the Special s»™«er Schools : 

An adv enture in living and learning for grades one thro ugh six. 
Chicago: The Board, 1963. 

Chicago Board of Education. Chicago Public Schools Special Elementary 
Summer S chools, 1967: A new dimen sion in growth. Chicago: The 

Board, 1967. 

Chicago Board of Education. The Special Elementary Summer Schools. 
Chicago: The Board. (Undated. - ) “ 

Youth Development Program 

Liddle, G. P. , Rockwell, R. E. , & Sacadat, E. Education improvement 
for the disadvantaged in an elementary setting . Springfield, 
Illinois: Charles C. Thomas, 1967* 

Matthews, C. V. A curriculum for drop-out prone students: Delinquency 

study and Youth Development Project . Edwardsville, Illinois: 
Southern Illinois University, 1966. 



IOWA 

Reading Project 

Reid, H. C. Preventive measures to reduce reading retardation in the 
primary grades. Cedar Rapids: Iowa State University of Science 

and Technology, 1966. 



KENTUCKY 

Head Start Program 



Ramsey , W. , & Boercker , M. The Influence of a Head Start program on 
reading achievement . Kentucky: University of Kentucky, 1967. 

LOUISIANA 

Education Improvement Project (EIP ) 

Plat tor, S. D. Preliminary findings from a longitudinal Educational 
I mprovement Project being conducted for Instructlonallv Impover- 
ished pupils In Intact schools in the urban South . Paper pre- 
sented to American Educational Research Association, 1968. 

New Orleans, Louisiana: New Orleans EIP, 1368. 

MARYLAND 

Collections of Evaluation Reports 

Baltimore City Public Schools, Special Projects and Programs. Special 
projects and programs . Baltimore, Maryland: The Schools, 1967. 

Baltimore City Public Schools, Special Projects and Programs. Summer 
1968 education al opportunities (for children, youth, and adults ). 
Baltimore, Maryland : The Schools, 1968. 

Specific Projects ' 

Cluster Concept Program 

Maley, D. An Investigation and development of the cluster concept as 
^ p rogram in vocational education at the secondary level . 

College Park: University of Maryland, 1966. 

Early School Admissions Program 

Baltimore City Public Schools. Progress report, 1962-1963: Early 

School Admissions Project . ""Baltimore , Maryland : The Schools, 1963. 

Baltimore City Public Schools. Progress report . .'963-1964: Early 

School Adm issions Project . Baltimore, Maryland: The Schools, 1964. 

B a ltimore City Public Schools. Early School Admissions Project : 

Promisin g practices . Baltimore, Maryland: The Project. (Undated.) 

Baltimore City Public Schools, Division of Personnel-ESEA. Parent 
liaiso n worker: Early School Admissions Program . Baltimore, 

Maryland: The Division. (Undated.) 

Early School Admissions Program. Guidelines . Baltimore, Maryland: 
Baltimore City Public Schools(?), 1964(?) 
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Early School Admissions Program. Parental involvement in Early School 
Admissions Child Development Centers . Baltimore, Maryland: 

Baltimore City Public Schools(?) (Undated.) 

Early School Admissions Program. The teacher aide. Baltimore, 

Maryland: Baltimore City Public Schools. (Undated.) 

Furno, 0. F. Culturally disadvantaged children: Pupil data >>ank . 

(Baltimore's Early School Admissions Project) Baltimore, Maryland: 
Baltimore City Public Schools, 1967. 

Head Start Project 

Conners, C. K. , & Eisenberg, L. The effect of teacher behavior on 

verbal intelligence in Head Start children . Baltimore, Maryland: 
Johns Hopkins University, School of Medicine, 1966. 

Waller, D. A., & Connors, C. K. A followup study of intelligence 
changes in children who participated in Project Headstart . 

Baltimore, 'Maryland: Johns Hopkins University, School of 

Medicine, 1966. 

Mission Project 

Baltimore City Public Schools, Bureau of Research. Research design for 

evaluating Project Mission . Baltimore, Maryland: The Schools, 1966. 

Project Mission Staff. A progress report « July 1. 196 6- June 30. 1967 . 
Baltimore, Maryland: Baltimore City Public Schools, Bureau "of 

Publications, 1967(?) 

Operation Moving Ahead 

Lynch, J. F. , Thomas, J. A., Shepherd, M. B. , & Mayor, B. The children's 
aide program in Operation: Moving Ahead . Upper Marlboro, Maryland: 

Prince George's County Board of Education. (Undated.) 

Prince George's County Board of Education. Final report (on) Operation : 
Moving Ahead, 1966 . Upper Marlboro, Maryland: The Board, 1966(?) 

MASSACHUSETTS 

Collection of Evaluation Reports 

Boston School Committee, Office of Program Development. Progress re- 
port (on several projects): March 1965-May 1966 . Boston, Massa- 
chusetts: Boston Public Schools, 1966. 

Specific Projects 

Elementary Enrichment Program (Counterpoise) 

Boston Public Elementary Schools. Operation Counterpoise: Init**! 

evaluation . Boston, Massachusetts: The Schools, .1964. 
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Boston Public Schools. Elementary Enrichment Program (Counterpoise) : 
Evaluation. 1965-1966. Boston, Massachusetts: The Schools, 1966. 

Boston Public bcnools. . Elementary Enrichment Program: Evaluation . 

1966-1967 . Boston, Massachusetts: The Schools, 1967. 

Model Demo n stration Subsystem Program 

Boston Public Schools, Office of Program Development, Developmental 
Schools Program. Appendix I to the assumptions, objectives and 
hypotheses document, being an attempt to define more precisely 
the ide al types of teacher and child behavior expected in a 
developm ental classroom . Boston, Massachusetts: The Program, 

1967. 6 

Clinchy, E. Schools jn the f ^riculum development in the 

urban school . (Prepared for the ESI Quarterly Report . Spring 1965.) 
Boston, Massachusetts: Educational Services, Inc., 1965. 

Clinchy, E. Magic, social studies, and the new progressivism . Paper 
delivered at the Cranbrook School, Bloomfield Hills, Michigan, 

February 13, 1965. Boston, Massachusetts: Educational Services, 

Inc. (Undated.) 



MICHIGAN 

Basic Reading Demonstration Project (BggP ) 

Detroit Public Schools, Research and Development Department. Sunary 
of project evaluation (ESEA Title I) : Evaluation of the Basic 

Reading Demonstration Project (BRDP) . Detroit, Michigan: The 

Department, 1968. 

Better Tomorrow for the Urban Child Program (BTU) 

Fairview Community School, Pre-Kindergarten, Kindergarten, and Tear One 
Staff. Getting ready for school . Flint, Michigan: Flint Com- 

munity Schools, 1968. 

Flint Board of Education. Better Tomorrow for the Urban Child (BTU) : 

The Mott Program of the. Flint Board of Education . Flint , 

Michigan: The Board, 1964(?) 

Board of Education, Research Services Department. Some preliminary 
data on the Title I program for 1966-67 . Flint, Michigan: The 

Department, 1967. 

Flint Community Schools, Research Services Department (?) Flint Comnunity 
Schools first year's progress report: Better Tomorrow for the 

Urban Child . Flint, Michigan: Flint Board of Education(?) , 1965(?) 

Flint Community Schools, Research Services Department (?) Flint Comnunity 
Schools Better Tomorrow for the Urban Child: Progress report for 

the third year of the program, 1966-67 . Flint, Michigan: The 

Department, 1967. 
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The Inter-School Reporter , 1968, 3(1). (Community education edition 
published by the Flint Connunlty Schools.) 

Great Cities School Improvement Project 

Detroit Public Schools. Great Cities School Improvement Project^ 

1966-67 . Detroit, Michigan: The Schools, 1967. 

Mitchell, C. Program s , """ i iry: The Detroit Great Cities School 

Improvement Project . Detroit, Michigan: The Project. (Undated.) 

Head Start Program (Summer) 

Detroit Public Schools, Research and Development Department. Summary 
of project evaluation (for pilot study on Summer Head Start) . 

Detroit, Michigan: The Department, March 1968. 

Preschool Child and Parent Education Project 

Detroit Public Schools, Research and Development Department. Summary of 
project evaluation (ESEA» Title I): Evaluation of the Preschool 

Child and Parent Education Project as expanded through the use of 
Elementary and Secondary Education Act, Title I, funds . Detroit, 

Michigan: The Department, 1968. 

Reading Instruction Project 

Hahn, H. T. A study of the relative effectiveness of three methods of 
teaching reading in grade one . Pontiac, Michigan: Oakland County 

Schools, 1965. 

Reading Program 

Whipple, G. Appraisal of the City Schools Reading Program . Detroit, 

Michigan: Detroit Public Schools, Division for Improvement of 

Instruction, Language Education Department, 1963. 

MINNESOTA 

Lincoln Learning Center 

Faunce , R. W. Lincoln Learning Center: An experimental junior high 

school of the Minneapolis Public Schools. Summary of evaluation 
results (for) the first two years, 1964-1966 . Minneapolis , Minne- 
sota: Special School District No. 1, 1967. 

Roffers, D. W. Final report: Lincoln Learning Center . ("Kids are 

our most important product.") Minneapolis, Minnesota: Minneapolis 

Public Schools, 1967. 

Youth Development Project 

Murton, B. J., Faunce, R. W. , & Neale, D. C. Project Motivation. 1964-65 : \ 

A Youth Development Project evaluation report . Minneapolis, Minnesota: 
Community Health and Welfare Council, 1966. 

Neale, C. , Proshek, J. M. , & Rundorff, R. L. Junior high orientation camp : 
Youth Development Project evaluation report . Minneapolis, Minnesota: 
Community Health and Welfare Council, 1966. 




*- 



88 



Kaygor, JJ. R. An evaluation of reading enrichment programs in three 
settlement house camps during the summer of 1965 (and) some 
notes on reading enrichment in a family camp * (Youth Develop- 
ment Project evaluation report.) Minneapolis, Minnesota: 

Community Health and Welfare Council, 1966. 

MISSOURI 

Lincoln Plus and Manual Plus Projects 

McFarland, R. Compensatory education: Statistical report. 1964-65 

(on testing program for Lincoln Plus and Manual Plus Projects) . 
Kansas City, Missouri: Department of Research and Development, 

1966. 

Youth Development Project 

Leake, D. B. , & Engle, G. A potential and actual school dropout pro- 
ject: A report on the Youth Development and Vocational Trai"*"p_ 

Program . St. Louis, Missouri: Presbytery of St. Louis, 1964. 

NEW JERSEY 

Enrichment Program 

Daniel, K. B., Cowles, M. , 6 Kay, C. A study of the effects of an 
enrichment program for disadvantaged junior high school pupils 
with respect to IQ scores . Paper read to American Educational 
Research Association, Chicago, February 19, 1966. Union, 

New Jersey: Newark State College, 1966. 

NEW MEXICO 

English Literacy Instruction for Preschool Indian Children 

Homme, L. Final report: A system for teaching English literacy to 

preschool Indian children . Washington, D. C.: U. S. Department 

of Health, Education, and Welfare, Office of Education, 1965. 

NEW YORK 

Collections of Evaluation Reports 

Evaluation of New York City Title I educational projects. 1966-67 . 

New York: Center for Urban Education, 1967. 

Madison Area Project Staff. Laboratory for change: The Madison Area 

Project . Syracuse, New York: City School District, 1964. 

Tieman , N • Evaluation of operation and effectiveness of s««mer school 
programs: Summer 1966 . Brooklyn, New York: Board of Education 

of the City of New York, Bureau of Educational Research, Office of 
Educational Research, 1967. 



Specific Projects 
ABLE Project (Buffalo) 



Margulis, J. D. Progress report for 1963-64 describing the Project ABLE 
program . Buffalo, New York: Board of Education, 1964. 

ABLE Project (Hartsdale) 

Waxman, S. M., Freilich, B. , Nemeth, G. , & Sindos, L. Project ABLE : 

Summary and evaluation of the five year program . Hartsdale, New 
York: Warburg Campus, Greenburgh School District No. 8, 1966. 

ABLE Project (New York City) 

Abramson, D. A. The effectiveness of full-time and coordinated guidance 
services in the high school: Project ABLE first annual report . 

New York: Board of. Education of the City of New York, Bureau of 

Curriculum Research and Bureau of Educational Program Research and 
Statistics, 1962. 

Reswick, J . The effectivess of full time and coordinated guidance services 
in the high school: Project ABLE second annua 1 report . New York: 

Board of Education of the City of New York, Office of Research and 
Evaluatior., Bureau of Educational Program Research and Statistics, 1963. 

Reswick, J. The effectiveness of full time and coordinated g^H dance 
services in the high school: Project ABLE third a""«^l report . 

New York: Board of Education of the City of New York, Office of 

Research and Evaluation, Bureau of Educational Program Research 
and Statistics, 1964. 

Reswick, J. The effectiveness of full time and coordinated guidance 
services in the high school: Project ABLE fourth annual report . 

New York: Board of Education of the City of New York, Bureau of 

Educational Program Research and Statistics, 1966. 

Reswick, J. Project ABLE: Effects of augmented guidance service for 

culturally disadvantaged high school students, 1961-1966 . New York: 
Board of Education of the City of New York, Bureau of Educational 
Program Research and Statistics, 1967. 

ABLE Project (Windsor) 

Donegan, D. I. The ABLE Program, 1966-1967 . Windsor, New York: 

Bureau of Guidance, 1967. 

Windsor Central School. An analysis of Project ABLE, 1961-1965 : 

Windsor Central School. Windsor. New York . Windsor: The School. 

(Undated.) 
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All-Day Neighborhood Schools Program 

Board of Education of the City of New York. Extended school services 
through the All-Day Neighborhood Schools . (Curriculum Bulletin 
No. 19, 1965-66 Series) New York: The Board, 1966. 

Sexton, P. An assessment of the All-Day Neighborhood Schools Program 
for culturally deprived children . New York: New York University. 

(Undated.) 

Auditory Perceptual Skills Training Program 

Feldmann, S., & Deutsch, C. P. A study of the t r fectiveness of train- 
ing for retarded readers in the auditory perceptual skills under- 
lying reading . New York: New York University, School of Educa- 

tion, Institute for Developmental Studies. (Undated.) 

Career Guidance Program 

Williams, T. Implementation of the Career Guidance curriculum and 
teacher training . New York: Center for Urban Education, 1967. 

Childrens Center 

Caldwell, B. M. , & Richmond, J. B. The Childrens Center: A micro- 

cosmic health, education,, and welfare unit . Syracuse, New York: 
State University of New York, Upstate Medical Center, Department 
of Pediatrics, Children* s Center, 1967. 

College Discovery and Development Program 

Board of Education of the City of New York. College Discovery and 
Development Program: Report on operations during the first 

year, 1965-66 . New York: The Board, 1966(?) 

Board of Education of the City of New York. Cc liege Discovery and 
Development Program: Prong II . New York: The Board, 1968. 

Brody, L. , Harris, B., & Lachica, G. Discovering and developing college 
potential of disadvantaged high school youth: A report of the 

second year of a longitudinal s '~.udy on the College Discovery and 
Development Program. New York: City University of New York, Divi- 

sion of Teacher Education, Office of Research and Evaluation, 1968. 

Tanner, D. , & Lachica, G. Discovering and developing the college 
potential of disadvantaged high school youth: A report of the 

first year of a longitudinal study on the College Discovery and 
Development Program . New York: City University of New York, 

Division of Teacher Education, Office of Research and Evaluation, 
1967. 
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Corrective Mathematics Program 

Grossman, A. S. Corrective mathematics services for d isadvantaged 
pupils in nonpublic regular day schools . New York: Center 

for Urban Education, 1967 . 

Corrective Reading Program 

Carton, A. S. Corrective reading services for disadvantaged pupils 
in nonpublic regular day schools . New York: Center for Urban 

Education, 1967. 

CRAFT Project 

Harris, A. J. , Coleman, M. , Serwer, B. L., & Gold., L. A continuation 

of the CRAFT Project: Comparing reading approaches with disadvan- 

taged urban Negro children in primary grades . New York. Assoc ia 
ted Educational Services Corporation, Selected Academic Readings 
Division, 1968. 

Harris, A. J. , & Morrison, C. The CRAFT Project; Final report of__a 
three-year project in teaching reading to disadvantaged urban 
Negro children. Paper presented at the International Reading 
Association Convention in Boston, 1968. New York: City Univer- 

sity of New York, Division of Teacher Education, Office of 
Research and Evaluation, 1968. 

Harris, A. J., & Serwer, B. L. Comparison of reading approaches in 
first-grade teaching with disadvantaged children . (The CRAFT 
Project) New York: City University of New York, 1966. 

Harris, A. J. , & Serwer, B. L. The importance of in structional time 
in classroom reading research: A further analysis of data from 

the CRAFT Project. New York: City University of New York, 

Division of Teacher Education, Office of Research and Evaluation. 
(Undated.) (Also appears in the Reading Research Quarterly , 1966, 
2, 27-56. 

Morrison, C. , & Harris, A. J. Grade equivalent comparis ons between 

disadvantaged Negro urban children with and withou t kindergarten 
experience when taught to read by several methods . (The CRAFT 
Project) New York: City University of New York. (Undated.) 

Demonstration Guidance Project 

Hillson, H. T., & Myers, F. C. The Demonstration Guidance Project , 
19 57-1962 . New York: Board of Education of the City of New 

York, 1963. 

Scully, M. M. The Demonstration Guidance Project and the teaching of 
English. In Improving English skills in culturally different 
youth in large cities. (U. S. Office of Education Bulletin No. 5) 
Washington, D. C. : U. S. Government Printing Office, 1964. 

Pp. 166-179. 



Wrightstone, J. W. , McClelland, S. D., & others. Assessm en t of _ t he 
Demonstration Guidance Project . New York: Board of Education 

of the City of New York, Bureau of Education, Division of 
Research and Evaluation, August 1965. 

Early Reading Project 

Martin, J. H. Freeport Public Schools experiment on early readin g 
using the Edison Responsive Environment Instrument . New York: 
Responsive Environment Corporation. (Undated.) 

English Project (Gateway English) 

Smiley, M. B. Summary report of a pilot year for Gateway English. 

New York: Hunter College, Project English Curriculum Study 

Center, 1965. 

Enrichment Program 

Goldstein, L. S. Evaluation of an enrichment p rogram for socially. 
disadvantaged children . New York: New York University, Insti- 

tute for Developmental Studies, 1965. 

Goldstein, L. S., & Deutsch, M. An enric hment program for socially 
disadvantaged children: Some preliminary findings . New York: 

New York University, Institute for Developmental Studies, 1968. 

Enrichment Program for Neighborhood Youth Corp s Enrollees 

Williams, E. B., & Tannenbaum, R. S. Educationa l enrichment for 
disadvantaged inschool Neighborhood Youth Corps enrollees 
during the summer (of) 1967 . New York: Center for Urban 

Education, 1967. 

Evening Guidance Centers Program 

Sebald D. D. Evening Guidance Centers for disadvan taged pupils of 

nubile and nonnubllc schools . New York: Center for Urban Educa- 

tion, 1967. 

Free Choice Open Enrollment Program 

Fox, D. J. Expansion of the Free Choice Ope n Enrollment Program* 

New York: Center for Urban Education, 1967. 

Thorndike, R. L. Free Choice Open Enrollment^ Junior high schools. 
New York: Center for Urban Education, 1966. 

Grade Reorganization Program 

Frankel, E. Grade reorganization of middle schools in the public^ 
schools system . New York: Center for Urban Education, 1967. 

Frankel, E. Grade reorganization preparator y to the establishment o f 
the four year comprehensive high school . New York: Center for 

Urban Education, 1967. 
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Head Start Program 



Chap lan y A. A. , & Platoff, J. Preschool child development program 
(Head Start) in disadvantaged areas of New York City; Summer 
1967 . New York: Center for Urban Education, 1967. 

Higher Horizons Program 

Hillson, H. T., & Myers, F. C. The Demonstration Guidance Project , 
1957-1962: Pilot program for Higher Horizons . New York: 

George Washington High School and the Board of Education of the 
City of New York, 1963. 

Wrights tone, J. W. Evaluation of the Higher Horizons Program for 
underprivileged children . New York: Board o I Education of 

the City of New York, Bureau of Educational Research, 1964. 

Wrightstone, J. W. , Forlano, G. , et al. Evaluation of the Higher 
Horizons Program for underprivileged children: A summary . 

New York: Board of Education of the City of New York, Division 

of Research and Evaluation, Bureau of Educational Research, 1965. 

History Instruction Project 

Edgar, R. W. The impact of learning and retention of specially 

developed history materials for culturally deprived children . 

New York: The Research Foundation of the City University. 

(Undated.) 

Improved School Services Program 

Steinhoff, C. R. Improved educational services in selected special 
service elementary and junior high schools . New York: Center 

for Urban Education, 1967. 

Inschool Guidance Program 

Sebald, D. D. Inschool guidance for disadvantaged pupils in nonpublic 
schools . New York: Center for Urban Education, 1967. 

MERCURY Project 

Carson, M. H. , Kaplan, B., & Stiller, A. Six-year summary report on 
Proiect MERCURY: A N ew York State talent search project under 

the National^efense Education Act (and) conducted at Madison 
High School April 1960-June 1964 including a graduate followup 
study June 1964- June 1965 . Rochester, New York: City School 

District(?), 1966(?) 

Municipal Cooperative Education and Work Program 

Hamburger, M. Report of the evaluation study of the Municipal Coopera- 
tive Education and Work Program . New York: New York City Depart 

ment of Personnel, April 1965. 
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Nursery School Experience and School Adjustment 



Goldstein, K. , & Chorost, S. A preliminary evaluation of nursery 
school experience on the later school adjustment of culturally 
disadvantaged children . New York: State University of New 

York, Down State Medical Center, 1966. 

Prekindergarten Program 

Schwartz, S. L. Preschool Child Development Centers in disadvantaged 
areas of New York City . New York: Center for Urban Education, 

1966. 

Schwartz, S. L. Expanded Prekindergarten Program . New York: Center 

for Urban Education, 1967. 

Preschool Enrichment Program 

Feldmann, S. C. A pre-school enrichment program for disadvantaged 
children. In F. Hechinger (Ed.), Preschool education today . 

New York: Doubleday, 1966. Pp. 97-104. 

Preschool Experience and Reading Achievement 

Wolf, M. , & Stein, A. Long range effect of preschooling on reading 
achievement . New York: Yeshiva University, Ferkauf Graduate 

School, 1966. 

School to Employment Program (STEP) (Buffalo) 

Board of Education. School to Employment Program: District progress 

report 1962-1963; 1963-1964 . Buffalo, New York: The Board. (Undated.) 

School to Employment Program (STEP ) (New York City) 

Meade, M. E. STEP School to Employment Program: School year 1963-1964 . 

New York: Board of Education of the City of New York, High School 

Division. (Undated.) 

Savitzky, C. STEP program /for potential dropouts. The Bulletin of 
The National Association of Secondary School Principals , 1963, 

47(287), 51-58. 

Science Instruction in Spanish 

Loretan, J. 0. Evaluation of science instruction in Spanish for students 
of Spanish-speaking background: Steps in implementing experimental 

project (for 1 school information ). Brooklyn: New York City Board of 

Education, 1964. 

Summer Institutes for Teachers 

Senf, R. Followup study of 1966 Summer Institutes for teachers of 
disadvantaged children. New York: Center for Urban Education, 

1967. 
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Summer Day Elementary School Program 



Cohen, H. Z., & Silbermintz, S. A. Report: Summer Day Elementary 

School Program . New York: Board of Education of the City of 

New York, 1968. 

Meiselman, M. S. Report: Summer Day Elementary School Program , 

July 5-August 15, 1967 . Brooklyn: Board of Education of 

the City of New York, Office of Elementary Schools, 1967. 

Summer Elementary School Program 

Fox, D. J., Shapiro, N. P., & Barnes, V. Summer 1967 Elementary 
School Program for disadvantaged pupils in poverty areas in 
New York City . New York: Center for Urban Education, 1967. 

Talent Search Project 

Brentwood Public Schools, Brentwood Guidance Department. Talent 
Search Project "Operation Challenge": Interim report for 

administrators and teachers . (Two-year findings on junior 
high school underachievers.) Brentwood, New York: The 

Department, 1964(?) 

Teacher Training Project (Bridge Project ) 

Downing, C. L. The preparation of teachers for schools in culturally 
deprived neighborhoods: The Bridge Project . Flushing, New York: 

City Universi*~ T of New York, Queens College, 1965. 

Greenberg, H. The Bridge Project followup study . Flushing, New York: 
City University of New York, Queens College, Division of Teacher 
Education, Office of Research Evaluation, 1965. 

Transitional Areas Program 

Kravetz, N. A special enrichment program of quality integrat ed 
education for schools in transitional areas . New York: 

Center for Urban Education, 1967. 

Tutorial Program 

Lohman, M. A. After-school tutorial and special potential development 
in I. S. 201 - Manhattan. New York: Center for Urban Education, 

1967. Pp. 18-20. 

Upward Bound Program 

Lachica, G., & Tanner, D. Upward Bound: The effects of an in-residence 

summer program on the academic year performance of underachieving 
disadvantaged high school youth . New York: City University of 

New York, 1967. 

Writing Instruction Project 

Center for Urban Education(?) A pilot project for testing linguistically 
oriented materials for the teaching of writing in New York City 
schools . New York: The Center, 1967. 
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NORTH CAROLINA 
Advancement School 



The Advancement School for, boys who could do better . Carnegie Quarterly, 
1966, 14(3) , 1-4. 

Learning Institute of North Carolina. The North Carolin a Advancement 
School, 1964-1967 . Durham: The Institute 1967. 

Education Improvement Program (EIP ) 

Cooper, G. (Ed.) Opening windows . Durham, North Carolina: Duke Univer- 

sity, Durham Education Improvement Program, 1968. 

Duke University, Durham Education Improvement Program. A study of the 
psvcholinguistic abil ities of eighty-nine culturally disa dvantaged 
children: A special study report on the ITPA . Durham, North 

Carolina: The Program, 1968. 

The Durham Education Improvement Program, 1966-1967 . Durham, North 
Carolina: Duke University, The Program, 1967. 

The Durham Education Improvement Program. 1966-1967: Research . 

Durham, North Carolina, Duke University, The Program, 1967. 

Friedlein, D. Tool technology for the classroom . Durham, North 
Carolina: Duke University, Durham Educational Improvement 

Program. (Undated.) 

Gallagher, J. J. Research and evaluation in the Educa tion Improvement 
Program: A report on discussions . Durham, North Carolina: Duke 

University, The Program, 1965(?) 

Turner, D. S. Sensory-motor activities for ear for childhood . Durham, 

North Carolina: Duke University, Durham Education Improvement 

Program. (Undated.) 

Programmed Instruction Project 

Long, E. R. , Jr. The effect of programmed instr uction in special skills 
during the preschool period on later ability patterns and academic 
achievement . Chapel Hill: University of North Carolina, 1966. 

Reading Instruction Project (Asheville^ 

Report to school personnel: Title I, ESEA Readin g Project (in_t|ie)^ 

Asheville City Schools . Asheville, Ncrth Carolina: The Schools, 1968. 

Robinson, R. E. First grade reading instructio n: Final report Con) 

Prolect 2874. Asheville, North Carolina: Asheville City Schools, 

1966. 

Robinson, R. E. Summary (of) Project 2874 : First grade reading 

instruction. Asheville, North Carolina: Asheville City Schools, 

1966. 
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Reading Instruction Project (Goldsboro) 



Bordeaux E. A. An evaluation of three ap proaches of teaching reading 
rfir^ gr ade . Raleigh, Nort h CaroUna: State Board of Educa- 

tion. Department of Public Instruction, 1966. 



OHIO 

Collections of Evaluation Reports 

Cincinnati Public Schools, Department of Instruction. Evaluation ^ 

the impact of the Elementary and Se c ondary Education Act, Tltle — ^ . 

Tn the Cincinnati Public S chools. Cincinnati, Ohio: e c oo , 

Cincinnati Public Schools, Department of Instruction. Education Act 
program evaluation. Journal of Instructiona l Research and_ Pro 

oram Development, 1967, 2(2); 3(1), (2), and (3). (Published 
occasionally by" the Cincinnati Public Schools.) 

Specific Projects 
Head Start Program 

Canton Public Schools(?) Report of Headstart testing P r opg » 

(Tables of test results.) Canton, 0hio(7): The scnoorsO), 1967. 




Young, B. W. A new approach to Head Start. Phi Delta Kappas ,. 1968, 
49(7), 386-388. 

Language Skills Improvement Project 

Wolf D. E. Report and ap p raisal: Project 181, improving 

’ skills for the educationally deprived . Montgomery County, Ohio. 

Jefferson Local Schools, 1966. 

uolf D E Re * appraisal: Pr otect 00340. Improving; language 

’ .Ml?; for the educationally deprived . Montgomery County, Ohio. 
Jefferson Local Schools, 1967. 
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PENNSYLVANIA 

Collection of Evaluation Reports 



Franklin Institute Research Laboratories, Systems Science Department. 
Title I (ESEA) in Philadelphia: The second year . (Parts A and B) 

Philadelphia: The Laboratories, 1967. 2 vol. 

Specific Projects 

Educational Improvement Program (EIP) 

Note: The bibliography for EIP is divided into. two subsections— 

(1) Materials re EIP administration and evaluation. 

(2) Materials re EIP subject-matter areas. 

(1) Materials yf EIP administration and evaluation . 

Efraemson, M. W. The chief consultant in the elementary Educational 
Improvement Program . Philadelphia: Philadelphia Public Schools, 

Elementary EIP, December 1966. 

Hayman, J. L. , Jr. An evaluation of Educational Improvement Program 
schools . Philadelphia: Philadelphia Public Schools, School 

District of Philadelphia, Division of Research, February 1967. 

Hayman, J. L., Jr. A report to the Beard of Education on findings of 

the U. S. Co— nission on Civil Rights on the Educational Improvement 
Program . Philadelphia: Philadelphia Public Schools, School Dis- 

trict of Philadelphia, Division of Research, February 1967. 

Johnson, M. S., and Kress, R. A. Philadelphia's Educational Improvement 
Program. Reading Teacher , 1965, 18, 488-492. 

Philadelphia Public Schools, School District of Philadelphia, Curricu- 
lum Office. The Educational Improvement Program: Brief summary 

of significant features . Philadelphia: The Office, August 1965. 

Philadelphia Public Schools, School District of Philadelphia, Curricu- 
lum Office, Educational Improvement Program, Great Cities School 
Improvement Program, Summer Workshop. An overview of the EIP . 
Philadelphia: The Summer Workshop, 1965. 

Philadelphia Public Schools, School District of Philadelphia, Division 

of Research. Evaluation of the elementary school Education Improve- 
ment Program, 1966-67 . Philadelphia, The Division, December 1966. 

Philadelphia Public Schools, School District of Philadelphia, Educa- 
tional Improver ent Program Office. The EIP . Philadelphia: 
Philadelphia Public Schools, School District of Philadelphia, 
Department of Superintendence, March 1964. 
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Philadelphia Public Schools, School District of Philadelphia, Elementa y 
Educational Improvement Program. The consulting teacher in the 
elementary EIP . Philadelphia: The Program, September 19b7. 

Philadelphia Public Schools, School District of Philadelphia, Elementary 
Educational Improvement Program. Job anal y sis for EIP consultin g 
teachers. Philadelphia: The Program. (Undated.) 

Philadelphia Public Schools, School District of Philadelphia, Office of 

infor^tional Services. An evaluation of (the) Ed u c ational Improve- 
ment Program . Philadelphia: The Office, November 1964. 

Philadelphia Public Schools, School District of Philadelphia, School- 
Community Coordinating Team. Great Cities Sc ; 

Program: Progress report, September 196 0- June 1962. Philadelph . 

The District, February 1963. 

Savitz, H. C. A week with a consultant j Come along vith gje. Philadelphia. 

Philadelphia Public Schools, Educational Improvement Program. 

(Undated.) 

cm-it-Vi F B et al. Services of classroom aides . Philadelphia: Phila- 

delphia* Pub lie Schools, School District of Philadelphia, Elementary 
Educational Improvement Program. (Undated.) 

(2) Materials re EIP subject-ma tter areas. 

Borish, E., Brown, P., & Renfrow, G. Individual phonics ggSSjgg * 

T.arh pr's guide for attaints tration. Philadelphia : Philadelphia 

Public Schools, School District^! Philadelphia, Office of Instruc- 
tional services, Elementary Educational Improvement Program, m '• 

Garber, C. N. Rational numbers . Philadelphia: Philadelphia Public Schools, 

School District of Philadelphia, Curriculum Office, Elementary 
Educational Improvement Program, January 1967. 

Garber, C. N. Sets. Philadelphia: Philadelphia Public Schools, School 

District of Philadelphia, Curriculum Office. (Undated.; 

Philadelphia Public Schools, School District of Philadelphia, Department 
of Curriculum, Elementary Educational Improvement Program. 12^. 
consulting teachers* list of favorite books for children. Philadel- 
phia: The Program, March 1967. 

Seamen, M., Welch, M., & Smith, F. B. Integrated language srts,yearon e: 
Big and little stores - Philadelphia: Philadelphia Public Schools, 

School District of Philadelphia, Department of Curriculum, El ry 

Educational Improvement Program, March 1967 reissue. 

c , f . « n Develooment of the la"p” a ge arts area . Philadelphia. Phila 

delphia Public Schools, School District of Philadelphia, Educationa 
Improvement Program, September 1966. 
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Smith F. R- The improvement of sp elling: Sp e lling and the Iowa Test s 

ofBasic Skills! Phil adelphia: Philadelphia Public Schools, School 

District of Philadelphia, Department of Curriculum, Elementary Educa 
tional Improvement Program, March 1967* 

Smith F. B. Usage and functional grammar . Philadelphia: Philadelphia 

Public Schools, School District of Philadelphia , Department oi Cur- 
riculum. Elementary Educational Improvement Program, March 1967. 

Smith, F. B. Introducing t he listening center, and questions tgJg SS ST 
oanv records! Philadelphia! Philadelphia Public Schools, School 
District of~ Philadelphia, Department of Curriculum, Elementary 
Educational Improvement Program, April 1967. 

Smith, F. B. Word recognition skills: CPP levels I to VII y *^lud ±gg 

listening activities and developing dictionary skills. Phila 
pSHs Philadelphia Public Schools, School District of Philadelphia, 
Office of Instructional Services, Elementary Educational Improvement 
Program, September 1967. 

Smith, F. B. Integrated language arts, yea, one: — Let's be friendl y. 
Philadelphia: Philadelphia Public Schools, School District of 

Philadelphia, Curriculum Office, Elementary Educational Improve- 
ment Program, January 1968. 

Smith, F. B., & Kravitz, I. The teaching of spelling. Philadelphia; 
Philadelphia Public Schools, School District of Philadelphia, 
Curriculum Office, Elementary Educational Improvement Program, 
October 1966. 

Taylor, N. L. Teaching elementary school mathematicsj — Organizin g the 
class and room, getting started (CPP level I) . Philadelphia : 
Philadelphia Public Schools, School District of Philadelphia, 
Curriculum Office, Elementary Educational Improvement Program, 

August 1966. 

Taylor, N. L. Teaching elementary schoo l mathematicsj — CPP level I _I. 
Philadelphia: Philadelphia Public Schools, School District of 

Philadelphia, Curriculum Office, Elementary Educational Improvement 

Program, November 1966. 

Taylor, N. L. Teaching of elementa r y school mathematicsj CPP level VI I, 
Parts 1 and 2. Philadelphia: Philadelphia Public Schools, School 

District of Philadelphia, Department of Curriculum, Elementary 
Educational Improvement Program, November 1966. 

Taylor, ». L. Some suggested activities f or the teaching of level .IV 

in ari thme tic. Philadelphia: Philadelphia Public Schools, School 

District of Philadelphia, Curriculum Office, Elementary Educational 
Improvement Program, December 1966. 
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Taylor, N. L. Teaching elementary school m athematics: — CPP level III , 

Parts 1 and 2. Philadelphia: Philadelphia Public Schools, School^ 

District of Philadelphia, Curriculum Office, Elementary Educational 
Improvement Program, December 1966. 

Taylor, N. L. Developing effective pro blem solving in the primary 
grades. Philadelphia: Philadelphia Public Schools ^School 

District of Philadelphia, Department of Curriculum, Elementary 
Educational Improvement Program, March 1967. 

Taylor, N. L. Number sentences or arithmetic with f rames . _ Philadelphia: 
Philadelphia Public Schools, School District of Philadelphia, 
Department of Curriculum, Elementary Educational Improvement Program, 

March 1967 reissue. 



Taylor, N. L. Some suggested act i vities for the teachi ”jj ° f 

school mathematics: CPP level V, Parts 1, 2^ . Philadelphia . 

Philadelphia Public Schools, School District of Philadelphia, 
Department of Curriculum, Elementary Educational Improvement 
Program, March 1967 reissue. 

Taylor N. L. Begi nning arithmetic . Philadelphia: Philadelphia Public 

Schools, School District of Philadelphia, Department of Curriculum, 
Elementary Educational Improvement Program, April 1967. 

Taylor, N. L. Some suggested activities for the teaching of element al 
mathematics: Level VI. Parts 1 and 2 . Philadelphia: Philadelphi 

Public Schools, School District of Philadelphia, Department of Cur- 
riculum, Elementary Educational Improvement Program, April 1967. 

Taylor, N. L. Parents can help children w ith mathematics: — Arithmetic 

is a social tool, a way of thin k ing; meaningful arithmetic is fu nc - 
tional and practical. Philadelphia: Philadelphia Public Schools, 

School District of Philadelphia, Elementary Educational Improvement 

Program. (Undated.) 

Ward, R. , & Smith, F. B. Integrate d language arts, year one: The fa rm. 

Philadelphia: Philadelphia Public Schools, School District of 

Philadelphia, Department of Curriculum, Elementary Educational 
Improvement Program, March 1967 reissue. 

Ward R.. Williams, C., S Smith, F. B. Integrated language arts, year _ o _pe: 
Time to wake up. Philadelphia: Philadelphia Public Schools, School 

District of Philadelphia, Department of Curriculum, Elementary Educa- 
tional Improvement Program, March 1967 reissue. 



Enrichment Program 

Khouri, J. Educational and cultural enrichment in selected schooj^, 

1967-1968. Pennsylvania: Bethlehem Area School District. (Undated.) 
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Language Instruction in a Head St art Prograia 

Rerzonskv M. D. An e valuation of an eight- w eek Head Start Program . 
Berzonsky , M. M _ - Beetlng o£ the Vrican Educational Res^rch 

Association, February 1967. Harrisburg: Pennsylvania Department 

of Public Instruction, Bureau of Research Administration and 

Coordination, 1967. 

Learning Centers Program 

Mattleman, M. S. An evaluation o f the effects of an enrichment program 
on six year old children . Philadelphia: Temple University, 1966. 



Preschool and Primary Education Project 



r_ A ,1. Summer school report, July 5, 1964-Aug u st ,14, 19 . 64 . 

Eke, Pennsylvania : School District of the' City of trie, WtfH 

Barber, G. A., et al. Annual report, 1964 -65: P re-Sd:ool and Primar y 

Education Project . Erie, Pennsylvania: School District of the 

City of Erie, 1965. 

narkor n A. . et al. Summer school report, July 11 t 19 65- August 13, 1965 . 
Pre-School and Primary EduTation Project . Erie, Pennsylvania: School 

District of the City of Erie, 1965. 

Residential Educational Project 

Mitchell, H. E. A summer pilot residential ed ucationa l pro j ect f or 

....ikrenv denrived youth . Philadelphia: University ot rennsyl- 

vania, 1965. 

Small Group Basic Instruction Program 

Gilbert, C. T. Small Group Basic Edu cation Program. Albion, Pennsylvania: 
Northwestern School District, Department of Public Instruction, 

July 1966. 

UPLIFT Project 

Forest Hills Review (Summer School Edition re Pioject UPLIFT) » 

August 1966. 

Smith, D. L. Evaluation progress report: Project UPLIFT— Readin g 

and English (in) junior and senior high school. Sidman, 

vania- Trilngle Area-Adams-Summerhill Joint Schools , August 1966. 



RHODE ISLAND 

Collection of Evaluation Reports 

Providence School Department. Providence schools, 1965-196 ^: 
on federal programs in the Providence School Department. 
Rhode Island: The Department, 1968 (?) 



Report 
Providence , 
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Specific Project 



Summer Kindergarten Program 

Johnston School Committee. Project report: Summ er Kindergarten Program, 

Title I ESEA. Johnston, Rhode Island: The Johnston Public Schools, 

1966. 

Johnston School Committee. Project report: Summer Kindergarten Pro gram, 

Title I ESEA. (Part I: Evaluation report .) Johnston, Rhode Island: 

The Johnston Public Schools, 1967. 

SOUTH CAROLINA 
Pre-Primary School Program 

gi—fr.r Pre-Primary School Program . Sumter, South Carolina: School 

District No. 17. (Undated.) 

Preschool Project in Individualized Reading Instruction 

Daniel, K. B., & Cowles, M. The effec t s of individualized instruc- 
tion on Head Start pupils* achievement . Paper read to the 
American Educational Research Association in Chicago, February 
1968. Columbia: University of South Carolina, 1968. 



TENNESSEE 

Early Training Project 

Gray, S. W. , & Klaus, R. A. Early Training Project: Interim report. 

Murfreesboro, Tennessee: City Schools and George Peabody College, 

November 1963. 

Gray, S. W. , & Klaus, R. A. An experimental pre school program for 

culturally deprived children . Montreal, Canada: American Associa 

tion for the Advancement of Science, December 1964. 

Gray, S. W. , & Klaus, R. A. An experimental preschool program for 
culturally deprived children. Child Development , 1965, 36(4), 
887-898. 

Klaus, R. A. , & Gray, S. W. Murfreesboro preschool program for culturally 
deprived children. Childhood Education , October 1965, 92-95. 

Klaus, R. A., & Gray, S. W. The Early Training Pr o ject for disadvantaged 
children: A report after five years . Nashville, Tennessee: George 

Peabody College, 1967. 
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First Grade Program for the Disadvantaged 

Brazziel, W. F. , & Terrell, M. For first-graders: A good start in 

school. The Elementary School Journal , April 1962, 352-355. 

Higher Ground Project 

Neely, M. D. , Bradley, L. D., & Dobbs, V. H. Evaluat ion report: 

Project Higher Ground 1966-67, a program of compensatory efforts 
for the educationally and culturally deprived of Metropolitan 
Nashville-Davidson County . Nashville, Tennessee: Metropolitan 

Schools of Nashville-Davidson County, 1967. 



TEXAS 

Bilingual Education Program 

Horn, T. D. A study of the effects of intensive ora l-aural English 
language instruction, oral-aural Spanish languag e instruction, 
and non-ora 1-aural instruction on reading read iness in grade one. 

Austin: University of Texas, 1966. 

Ott, E. Bilingual Education Program . Austin, Texas: Southwest 

Educational Development Laboratory. (Undated.) 

Ott, E. Bilingual research study: Linguistic build ups in English 

for disadvantaged Spanish speaking children . Austin: University 

of Texas, Research and Development Center. (Undated.) 

Head Start Program 

El Paso Public Schools, Department of Testing and Psychological Services. 
Headstart III a nd IV in the El Paso Public Schools: An evaluative 

study . El Paso, Texas: The Department, 1967. 

Language Development of Head Start Pupils 

Daniel, A. A., & Giles, D. A comparison of the oral language development 
of Head Start pupils with non-Head Start pupils . Fort Worth(?) , 

Texas: Dunbar Elementary School(?), August 1966. 

Language Stimulation Program 

Carter, J. L. The long range effects of a lan guage stimulation program upo n 

Negro educationally disadvantaged first grade children: Final report. 

Houston, Texas: Houston University, 1967. 

Reading Project 

El Paso Public Schools, Division of Instruction. Remedial reading summar y. 
El Paso, Texas: The Division, May 1968. 



105 



VIRGINIA 
Berkeley Project 



Wells, H. R. The Berkeley Project . Norfolk, Virginia: Berkeley 

Project Schools, April 1967. 

Higher Horizons Program 

Brazziel, W. F. Higher Horizons in southern elementary schools. 
Journal of Negro Education , 1964, 33 , 382-389. 

Brazziel, W. F. Head start on a new kind of life: An assessment of 

gains in two summer programs. Integrated Education , 1966, 4^ 
42-46. 

Brazziel, W. F., & Gordon, M. Replications of some aspects of the 
Higher Horizons Program in a southern junior high school. The 
Bulletin , 1963, 47 (No. 281), 135-143. 

Brazziel, W. F., & Terrell, M. An experiment in the development of 
readiness in a culturally disadvantaged group of first grade 
children. Journal of Negro Education , Winter 1962, 31 , 4-7. 

Human Development Project 

Richmond Public Schools, Human Development Project report, 1965-66 . 
Richmond, Virginia: The Schpols, 1966. 

WASHINGTON 

Special Summer School Program 

Special Summer School, 1966. Seattle, Washington: Seattle Public 

Schools, 1966. 

\ 

Special Summer School, 1967 . Seattle, Washington: Seattle Public 

Schools, 1967. 

WEST VIRGINIA 

Preschool Program 

Hardy County Schools. Pre-School Program in Hardy County, West 
Virginia under the Office of Economic Opportunity. Community 
Action Program: Guidebook for teachers . Hardy County : The 

Schools, 1965. 
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WISCONSIN 

Collection of Evaluation Reports 



Milwaukee Public Schools. Summary evaluation of Title I Elementary 
and Secondary Education Act projects, 1966-1967 . Mi lwaukee , 
Wisconsin: The Schools, 1967 (?) 

Specific Projects 

Individualized Mathematics Instruction Program 

Ashbaugh, W. H. , Rowe, G. D., Belton, J. , & Sanders, R. L. A program 
to provide individualized instruction for junior high school 
mathematics students: September 6, 1966- June 16, 1967 . Milwaukee, 

Wisconsin: Milwaukee Public Schools, Division of Curriculum and 

Instruction, 1967 ( ? ) 

Kindergarten Program 

Larson, R. G. , & Olson, J. L. Final report: A pilot project for 

culturally deprived kindergarten children . Racine, Wisconsin: 

Unified School District No. 1, 1965. 

Language Arts Program 

Ashbaugh, W. H. , Rowe, G. D., Belton, J. , & Sanders, R. L. Secondary 

English language arts: September 6, 1966- June 16, 1967 . Milwaukee , 

Wisconsin: Milwaukee Public Schools, Division of Curriculum and 

Instruction, 1967(?) 

Reading Instruction Project 

MacDonald, J. B. An experimental study of the group versus the one-to- 

one instructional relationship in first grade basal reading programs . 
Madison: University of Wisconsin, School of Education, 1966. 

Remedial Teachers Program 

Ashbaugh, W. , Rowe, G. D., Belton, J., & Keyser, J. Special remedial 
teachers in elementary school basic skills: September 6. 1966- 

June 16, 1967 . Milwaukee, Wisconsin: Milwaukee Public Schools, 

Division of Curriculum and Instruction, 1967 (?) 

Research and Instruction Units Program (R&I Units ) (Madison) 

Klausmeier, H. J. , & Quilling, M. (Eds.) Technical report No. 48 : 

Research and development activities in R&I units of four elemen- 
tary schools of Madison, Wisconsin, 1966-67 . Madison: University 

of Wisconsin, Wisconsin Research and Development Center for Cognitive 
Learning, 1968. 
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Research and Instruction Units Program (R&X Units) (Milwaukee ) 

Quilling, M. , Cook, D. M. , Wardrop, J. L., & Klausmeier, H. J. (Eds.) 
Technical report No. 46: Research and development activities 

in R&I units of two elementary schools of Milwaukee, Wisconsin , 
1966-67 . Madison: University of Wisconsin, Wisconsin Research 

and Development Center for Cognitive Learning, 1968. 

Research and Instruction Units Program (R&I Units) (Racine) 

Klausmeier, H. J., Quilling, M. , & Wardrop, J. L. (Eds.) Technical __ 
report No. 52: Research and development activities in R&l units 

of five elementary schools of Racine, Wisconsin, 1966-67 . Madison: 
University of Wisconsin, Wisconsin Research and Development Center 
for Cognitive Learning, 1968. 
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APPENDIX B 



March 29, 1968 G-52 



DESIGN RATING ITEMS 

1. The population is described 

2. The sampling procedures are fully Ow&cribed . 

3. Objectives are clearly specified in detail. . 

4. The criteria are clearly specified ■ 

5. A list is given of the variables considered 



6. Measuring instruments used are directly related to the objectives 

7. Measuring instruments used are either »ell-known or fully described . . . 

8. Data are recorded in tables . . 



9. The types of statistical analyses employed are described 

10. Significance (P) levels are quoted 




APPENDIX C 



LIST OF CONSULTANTS 



Dr. Thomas Fox 

Office of the Assistant Secretary for Program Evaluation 
U. S. Office of Education 
Washington , D. C. 

Dr. Mildred Fitzpatrick 
Chairman, Title I 
State Department of Education 
Santa Fe, New Mexico 

Dr. Charles Hammer 
Division of Compensatory Education 
U. S. Office of Education 
Washington, D. C. 

Dr. Robert Hess 
Professor of Education 
Stanford University 
Stanford, California 

Dr. Earl Jones 
Director 

Navy Training Research Laboratory 
San Diego, California 

Dr. James Jones 
Professor of Education 
New York University 
New York, New York 

Dr. Richard Lawrence 

National NDEA Institute for Advanced Studies in Teaching 
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